Newest at the top
2025-07-28 17:43:31 +0200 | trickard_ | (~trickard@cpe-51-98-47-163.wireline.com.au) |
2025-07-28 17:43:18 +0200 | trickard_ | (~trickard@cpe-51-98-47-163.wireline.com.au) (Read error: Connection reset by peer) |
2025-07-28 17:41:37 +0200 | Unicorn_Princess | (~Unicorn_P@user/Unicorn-Princess/x-3540542) Unicorn_Princess |
2025-07-28 17:37:41 +0200 | AVA | (~AVA@84.54.80.216) (Read error: Connection reset by peer) |
2025-07-28 17:28:31 +0200 | <Henson> | merijn: yes |
2025-07-28 17:28:13 +0200 | <merijn> | ok, so you do specifically need malloc :) |
2025-07-28 17:28:05 +0200 | gmg | (~user@user/gehmehgeh) gehmehgeh |
2025-07-28 17:27:48 +0200 | <Henson> | merijn: the problem with that is that in my production code I don't know a-priori the amount of memory that should be allocated, because it's being allocated in the C/C++ layer and passed back to Haskell. |
2025-07-28 17:26:43 +0200 | <merijn> | https://hackage.haskell.org/package/base-4.21.0.0/docs/Foreign-ForeignPtr.html#v:mallocForeignPtrB… |
2025-07-28 17:26:40 +0200 | <merijn> | I've always used Foreign.ForeignPtr, so I was guessing that's why I'm used to seeing it in profiles while you might not |
2025-07-28 17:25:51 +0200 | <merijn> | Since it uses GHC's newAlignedPinnedByteArray, rather than malloc |
2025-07-28 17:25:16 +0200 | <merijn> | Henson: I meant more like, using mallocForeignPtrBytes, which allocates using GHC's allocator and SHOULD appear in memory profiles |
2025-07-28 17:24:41 +0200 | <Henson> | merijn: the problem also happens with C++ "new" and "delete" |
2025-07-28 17:24:13 +0200 | trickard_ | (~trickard@cpe-51-98-47-163.wireline.com.au) |
2025-07-28 17:24:11 +0200 | <merijn> | Henson: Do you specifically need C malloc or just any allocation? |
2025-07-28 17:23:58 +0200 | <Henson> | merijn: oh, also, the leak when using "async" will not happen with "+RTS -N1" |
2025-07-28 17:23:43 +0200 | trickard | (~trickard@cpe-51-98-47-163.wireline.com.au) (Ping timeout: 245 seconds) |
2025-07-28 17:23:34 +0200 | <merijn> | (not your leak problem, the profiling one) |
2025-07-28 17:23:23 +0200 | <merijn> | The problem is calloc :) |
2025-07-28 17:23:17 +0200 | <merijn> | Henson: Ah, wait |
2025-07-28 17:22:40 +0200 | <Henson> | merijn: I don't think it does. I've done every kind of memory profiling the GHC's profiling offers, and the C-based memory allocation doesn't show up. |
2025-07-28 17:22:40 +0200 | sam113102 | sam113101 |
2025-07-28 17:22:36 +0200 | sam113101 | (~sam@modemcable200.189-202-24.mc.videotron.ca) (Read error: Connection reset by peer) |
2025-07-28 17:22:33 +0200 | sam113102 | (~sam@modemcable200.189-202-24.mc.videotron.ca) sam113101 |
2025-07-28 17:21:03 +0200 | <merijn> | I would expect that to still show up as a PINNED allocation |
2025-07-28 17:21:03 +0200 | <Henson> | merijn, magic_rb: I'm working on putting together a minimal reproducer. |
2025-07-28 17:20:15 +0200 | <Henson> | merijn: I have tried profiling, but the problem is that the calloc returns a pointer, which is tiny, but points to a large amount of memory. Because the memory is allocated in the C level, the RTS isn't able to keep track of it, so no significant memory usage shows up. |
2025-07-28 17:19:21 +0200 | <merijn> | That said, this is hard to troubleshoot without a minimal reproducer like magic_rb said |
2025-07-28 17:18:31 +0200 | <merijn> | Or checked the GC report? |
2025-07-28 17:18:00 +0200 | <merijn> | Henson: Have you profiled yet? |
2025-07-28 17:17:30 +0200 | <Henson> | merijn: yes, just to "cancel" them at the end. If I don't keep the async values then the problem still occurs. The memory usage also seems to follow a pattern of flat usage for several seconds, then an increase in about 2 seconds to a higher value, then it flattens for maybe 7 seconds, then another gradual increase, etc. |
2025-07-28 17:16:08 +0200 | <merijn> | Not to mention sandboxes involved tedious manual management |
2025-07-28 17:15:39 +0200 | fp | (~Thunderbi@2001:708:20:1406::10c5) (Ping timeout: 260 seconds) |
2025-07-28 17:14:29 +0200 | <dolio> | In the old sandbox setup, each (transitive) dependency would only ever be built once per sandbox. But there was no sharing of common dependencies between sandboxes, even if they wanted identical build specifications. |
2025-07-28 17:13:23 +0200 | <merijn> | Henson: The question is: Are you keeping Async values around somewhere? |
2025-07-28 17:10:45 +0200 | lortabac | (~lortabac@2a01:e0a:541:b8f0:55ab:e185:7f81:54a4) (Quit: WeeChat 4.5.2) |
2025-07-28 17:09:35 +0200 | <dolio> | And the analogy with cabal is that common dependencies of multiple projects are built at most once and shared, rather than being built once for each project. |
2025-07-28 17:09:30 +0200 | kuribas | (~user@ptr-17d51epjby9myrcgbo7.18120a2.ip6.access.telenet.be) (Ping timeout: 244 seconds) |
2025-07-28 17:09:05 +0200 | <Henson> | the leak seems to happen equally with GHC 8.10.7 and 9.12.2 |
2025-07-28 17:08:28 +0200 | <Henson> | merijn: I also tried using forkIO and forkOS instead of async, and the problem still happens. |
2025-07-28 17:08:10 +0200 | <Henson> | merijn: the creator/destructor threads are created in pairs and use an MVar to pass the memory pointers back-and-forth. Only 10 creator/destructor pairs are ever created, but many many memory allocations/deallocations are performed. When the memory block allocated is a multiple of 1024*1024 (1 MB) then the leak also doesn't happen. Based on the block size the leak can be faster or slow |
2025-07-28 17:07:42 +0200 | <dolio> | CSE is 'common subexpression elimination.' It's a transformation where you look for subexpressions that occur multiple times, then turn them into named bindings so that they are evaluated at most once. |
2025-07-28 17:03:29 +0200 | remmie | (ianremsen@tilde.team) remsense |
2025-07-28 17:00:46 +0200 | trickard_ | trickard |
2025-07-28 16:55:58 +0200 | vulpine | (xfnw@user/meow/xfnw) xfnw |
2025-07-28 16:55:53 +0200 | LainIwakura | (~LainIwaku@user/LainIwakura) (Ping timeout: 272 seconds) |
2025-07-28 16:48:25 +0200 | AVA | (~AVA@84.54.80.216) |
2025-07-28 16:40:31 +0200 | Lycurgus | (~juan@user/Lycurgus) (Quit: irc.renjuan.org (juan@acm.org)) |
2025-07-28 16:38:08 +0200 | <merijn> | So if you're holding on to async's the threads don't get GCed and (in turn) nothing in that thread's GC root will be |
2025-07-28 16:37:41 +0200 | <merijn> | Async contains a ThreadId and threads are not GCed while you hold on to their threadid |