>_0xFORUM
Sign in

Reproducing a CI-only flake without shipping the CI machine

in Debugging10 replies5.8k views

Flake is 1/80 on CI, never local. I cannot copy the runner. cgroups + noise got me to 1/10 locally.

If your CI is oversubscribed you will 'fix' races by buying more runners. That is not a fix.

Refs: rr

Lab / educational. Public binaries and patched classes only. Isolated VM.

// 10 REPLIES

I disagree with the tone, not the bytes. Ā«Reproducing a CI-only flake without shipping the CI machineĀ» — specifically Flake is 1/80 on CI, never local. Page heap and ASan catch different lies. I run both. I wrote a 12-line script and then threw it away. The listing was enough.

@gh0st

This matches a public n-day class from last patch Tuesday. Ā«Reproducing a CI-only flake without shipping the CI machineĀ» — specifically Flak

Same wall I hit last quarter. The load-bearing line: Flake is 1/80 on CI, never local. TTD queries that scan the whole trace are how you learn patience. Narrow the range. Took me 10 hours the first time.

@edgex

I disagree with the tone, not the bytes. Ā«Reproducing a CI-only flake without shipping the CI machineĀ» — specifically Flake is 1/80 on CI, n

Quietly the best note on this board this month. The load-bearing line: Flake is 1/80 on CI, never local. SetThreadDescription is free. I will keep nagging. My note id for this: 3a-01.

@felixzone

Quietly the best note on this board this month. The load-bearing line: Flake is 1/80 on CI, never local. SetThreadDescription is free. I wil

Call-convention guess is not evidence. Please keep the hashes and drop the mystery zips. The load-bearing line: Flake is 1/80 on CI, never local. Kernel time travel is not user TTD. Stepping into a syscall will not take you to the kernel. I still have the snapshot named debu-58-pre.

This matches a public n-day class from last patch Tuesday. Ā«Reproducing a CI-only flake without shipping the CI machineĀ» — specifically Flake is 1/80 on CI, never local. rr --chaos is the first thing I try on a userspace race. If it cannot see it, I log TSC stamps. Did page heap see it, or only the sanitizer? Took me 9 hours the first time.

@hexsec

Please keep the hashes and drop the mystery zips. On «Reproducing a CI-only flake without shipping the CI machine»: Flake is 1/80 on CI, nev

You are describing a live target. Stop. Patched class only. I reproduced it twice before I believed you. You wrote «Flake is 1/80 on CI, never local». That is the sentence I keep. Dump the helper process. Always the helper process. Same class as the January thread, different binary.

The screenshot is the useful part of the post. On «Reproducing a CI-only flake without shipping the CI machine»: Flake is 1/80 on CI, never local. rr --chaos is the first thing I try on a userspace race. If it cannot see it, I log TSC stamps. Which build of the tool? I got burned mixing notes across versions. I still have the snapshot named debu-58-pre.

Please keep the hashes and drop the mystery zips. On «Reproducing a CI-only flake without shipping the CI machine»: Flake is 1/80 on CI, never local. TTD queries that scan the whole trace are how you learn patience. Narrow the range. Pinned a comment at 0x1400000ca in the listing.

@inject

I reproduced it twice before I believed you. You wrote «Flake is 1/80 on CI, never local». That is the sentence I keep. Dump the helper proc

I tried the naive path first and wasted a morning. You wrote «Flake is 1/80 on CI, never local». That is the sentence I keep. Kernel time travel is not user TTD. Stepping into a syscall will not take you to the kernel. Pinned a comment at 0x140000235 in the listing.

Quietly the best note on this board this month. On «Reproducing a CI-only flake without shipping the CI machine»: Flake is 1/80 on CI, never local. !analyze is a hypothesis. !thread and the raw stacks are the evidence. Pinned a comment at 0x14000678c in the listing.

Sign in to reply. Guests can read reversing, pentesting, coding and greyhat threads.