logo FEDIDEVS POSTS

Frameworks, Libraries, and others

A survey of deep learning for mathematical reasoning. ~ Pan Lu, Liang Qiu, Wenhao Yu, Sean Welleck, Kai-Wei Chang. aclanthology.org/2023.acl-long

2 2 0

T.I.L. There was a genus of trilobites called Walliserops that had a mysterious trident shaped appendage growing from each of their heads.

2 1 0

@Edent it’s as if they intentionally make it as vague as “on the phone”

If the purpose is to identify a caller, why can’t they specifically ask “Have you dialed our number?”

2 0 0

@tychotithonus "find out how selinux works" has just been restored to my todo list.
The standard unix permission system is pretty coarse grained and kind of built on the assumption that programs only do what the person who launched them wanted them to do.

2 0 0

I greatly appreciate my computer having things like an MMU and an operating system but that's not going to stop me occasionally spinning up an Apple ][e emulator or having an absolute blast when I do. (Having friends who had Commodore 64's was also a pretty salient part of my childhood but I can't remember the names of any of their games)

2 1 0

The quest for a working file contents retrieval (followed by processing and file removal) over continues. As planned, implemented it with ( curl bindings), mostly combined with FTP, using that custom glob pattern matching function. Instead of using a dedicated thread for traversal, implemented caching shared across threads. As SSHFS, it seemed to work for some time, but then I noticed odd curl errors here and there (CurlFailedInit, for instance), and the growing memory consumption.

Upon further poking with a basic test program, both on the server with an old system (and an old curl version) and locally with a newer one, found that there are more errors when it runs in many threads, and especially if it is used with the threaded GHC runtime, but they happen with a single thread as well. Downloading the same file over and over, noticed that its downloaded size is wrong sometimes. And its memory consumption keeps growing, in all the scenarios, though maybe a little less on the newer system and without the threaded GHC runtime. With HTTPS in a single thread, too, on both systems. Same with curlGetString_ instead of do_curl_. It seems to use more or less constant memory when String is used though, while appears to leak memory with ByteString (lazy or strict) and [ByteString] (strict).

CurlBuffer instances use peekCStringLen for String, and packCStringLen for the other options, but I think they should behave about the same. Tried to profile it (built for profiling, then +RTS -p -hy), and the memory consumption did not grow while profiled, making it a heisenbug. That suggested that the issue may be simply with the GC not being aggressive enough, and +RTS -A5m (after building with -rtsopts) fixes it. Not sure if the GC will catch up before OOM killer gets it if I will run it without those RTS options. Though I am unsure why -A helped with that, and why profiling did, but likely both of them affect the garbage collector's behavior.

There are still libssh2 bindings to try, if this will not work well. Or simply to try and see if it will produce fewer odd errors and less of strange behavior.

Update: +RTS -M50m (simply limiting the heap size) fixes the "leak" as well; the GC can handle it, but for some reason chooses not to by default. At least it is not a "real" memory leak, apparently.

2 1 1

@dstndstn

Err, I may have been saying your name wrong then all this time then...

2 0 0