![]() ![]() The team quickly mitigated the problem, and gave me a nudge that I might want to look into the problem. If you’ve worked with large volumes of data for a long time, you won’t even bat an eyelash at the idea that a data store somewhere contains some really weird records.) The culprit turned out to be JSON strings containing zillions of backslashes. These days, the Haskell-powered Sigma uses aeson, the JSON library I wrote, to handle JSON data.Ī few months ago, the Haxl rewrite of Sigma was going through an episode of crazytown, in which it would intermittently and unpredictably use huge amounts of CPU and memory. To understand my accidental bit part in the show, let's begin by mentioning that Sigma uses JSON internally for various purposes. Nevertheless, success brings with it surprises, and even though I haven’t worked on Sigma or Haxl, I’ve been implicated in one such surprise. Throughput is higher than under its predecessor, and CPU usage is lower. The rewrite of Sigma in Haskell, using the Haxl library that Simon Marlow developed, has been a success. Sigma has a mission-critical job, and it needs to scale: its growing workload currently sees it handling tens of millions of requests per minute. Over the past few months, the Sigma engineering team at Facebook has rolled out a major Haskell project: a rewrite of Sigma, an important weapon in our armory for fighting spam and malware.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. Archives
June 2023
Categories |