merijn 2017-02-27 10:45:59
ertes: Baically, I wanna benchmark Chan/TChan, etc. performance, but I'm not happy with the existing benchmarks of, e.g. unagi-chan. They're rather simplistic...
ertes 2017-02-27 10:46:45
merijn: well, criterion's main contribution is the statistical stuff (and some care not to optimise the benchmark away)
ertes 2017-02-27 10:47:01
on IO you don't have to be that careful, and the statistical stuff you could do yourself using the 'statistics' package
merijn 2017-02-27 10:47:12
but there's two problems with criterion: 1) how do I prevent chan contents from being prematurely GCed (which a cleanup would handle), and how do I allocate a new Chan for every test (avoiding build up and heap expansion)
merijn 2017-02-27 10:47:39
ertes: I like the pretty html criterion generates, though!
ertes 2017-02-27 10:47:58
is building a Chan that expensive? i would expect it to be on the order of a newTVar
merijn 2017-02-27 10:48:20
ertes: No, but you need to do other setup, i.e. pre-launching threads for concurrent reader/writer tests
ertes 2017-02-27 10:48:20
so it should barely make a difference, if your actual benchmark is complex enough
merijn 2017-02-27 10:48:48
ertes: My benchmark is just, concurrent reads, writes, reads+writes for different numbers of readers and writers
ertes 2017-02-27 10:49:09
merijn: you could use a TVar (Chan …)
ertes 2017-02-27 10:49:33
then you can create the threads outside of defaultMain and just change the channel
merijn 2017-02-27 10:49:35
because e.g., unagi-chan is only benchmarking equal numbers of readers and writers and directly reading and writing in the same thread
merijn 2017-02-27 10:49:57
ertes: If you don't empty the Chan in between benchmarks your heap explodes and so does your GC time
merijn 2017-02-27 10:50:11
ertes: Which means later benchmarks become increasingly unreliable
ertes 2017-02-27 10:50:34
merijn: i mean: create a new TChan, then update the TVar to let the pre-existing threads use the new TChan
ertes 2017-02-27 10:50:53
but yeah, GC might be noticable
lyxia 2017-02-27 10:50:57
dolio: is it fine that F is contravariant?
dolio 2017-02-27 10:51:31
Yes, it would just be contravariant map. @free assumes everything is covariant.
dolio 2017-02-27 10:52:00
And adding the second parameter would just add more things you can do to it that you don't care about.
dolio 2017-02-27 10:52:08
(And @free doesn't support two parameters.)
merijn 2017-02-27 10:52:21
ertes: that introduces STM overhead even for the non-STM stuff too
ertes 2017-02-27 10:52:38
merijn: you can use an IORef instead of a TVar =)
ski 2017-02-27 10:52:44
@free mirror :: Either a b -> Either b a
lambdabot 2017-02-27 10:52:45
$map_Either g f . mirror = mirror . $map_Either f g
ertes 2017-02-27 10:53:02
merijn: it's the best solution i can think of without patching criterion
dolio 2017-02-27 10:53:08
Oh, I guess it supports some two parameter stuff.
ertes 2017-02-27 10:53:26
merijn: what you could do is to do the benchmarking code yourself, but then generate a criterion report and use its reporting functions
ski 2017-02-27 10:53:44
@free sigma :: F a b -> F b a
lambdabot 2017-02-27 10:53:44
Plugin `free' failed with: Sorry, this type is too difficult for me.
ski 2017-02-27 10:53:47
oh
dolio 2017-02-27 10:53:51
Yeah.
merijn 2017-02-27 10:54:04
ertes: I'm ok with patching criterion and submitting that upstream, I just want to do it with minimal work ;)
ski 2017-02-27 10:54:29
@free inverse :: (a -> b) -> (b -> a)
lambdabot 2017-02-27 10:54:30
g . h = k . f => f . inverse h = inverse k . g
ertes 2017-02-27 10:54:52
merijn: i think nobody would object to virtual time… it should be easy enough to do, too =)
ertes 2017-02-27 10:55:09
do linear regression on the time that was actually spent in the benchmark
ertes 2017-02-27 10:55:16
you can do the same with cpuTime
merijn 2017-02-27 10:56:17
ertes: I think it might be trickier than you think
ertes 2017-02-27 10:56:25
all you need is to wrap the benchmark in a with* sort of function that starts the timer, and at the end updates an IORef that accumulates the total
merijn 2017-02-27 10:56:52
ertes: Currently it runs a benchmark N iterations, and takes the total time of that. But now you need to also track N cleanup/setup times within that and store those
ertes 2017-02-27 10:57:15
no, the idea is precisely *not* to track those
merijn 2017-02-27 10:57:53
ertes: Wouldn't the IORef update introduce too much overhead for some of the small benchmarks people use it for?
ertes 2017-02-27 10:58:11
merijn: the IORef update isn't itself timed
ertes 2017-02-27 10:58:52
right now criterion needs to ask the clock once after each iteration
ertes 2017-02-27 10:59:00
with this idea implemented it needs to ask twice per iteration
ertes 2017-02-27 10:59:09
one of those should be untimed
dolio 2017-02-27 10:59:10
I don't think criterion does that.
ertes 2017-02-27 10:59:24
dolio: then how does it detect outliers?
dolio 2017-02-27 10:59:53
Hmm.
ertes 2017-02-27 10:59:55
how does it even calculate the standard deviation?
ertes 2017-02-27 11:00:06
it has to ask the clock between iterations
merijn 2017-02-27 11:00:43
ertes: No
merijn 2017-02-27 11:00:53
ertes: criterion does not request a clock after each iteration
dolio 2017-02-27 11:01:28
I'm skeptical that it can do that, because the clocks wouldn't be precise enough to time things.
merijn 2017-02-27 11:01:37
oh, wait, it does, I think?
ertes 2017-02-27 11:01:43
it has to
merijn 2017-02-27 11:01:47
dolio: Not true
ertes 2017-02-27 11:01:52
regression needs time samples
merijn 2017-02-27 11:01:54
dolio: It's using FFI to use rdtsc
dolio 2017-02-27 11:02:01
Oh.
merijn 2017-02-27 11:02:04
dolio: Which is super cheap
merijn 2017-02-27 11:02:44
ertes: Actually, both right
dolio 2017-02-27 11:02:53
That makes some of its design even more annoying.
merijn 2017-02-27 11:02:54
ertes: So it runs N batches of M samples
merijn 2017-02-27 11:03:16
And only times the M samples in aggregate
ertes 2017-02-27 11:03:25
yeah, that sounds reasonable… but if you want to subtract initialisation/cleanup, then you need to do it for each iteration
ertes 2017-02-27 11:03:51
in fact you need to ask the clock multiple times
ertes 2017-02-27 11:04:09
but only one of those should affect the time on average
merijn 2017-02-27 11:04:52
Actually, this is workable for me
merijn 2017-02-27 11:05:12
oh, wait...no, only for the simple case
merijn 2017-02-27 11:08:58
Can't I sucker someone into writing this for me so I don't have to bother? xD
ertes 2017-02-27 11:10:47
merijn: i'm happy to write the timing code for you, but i'll leave the criterion integration to you =)
merijn 2017-02-27 11:12:15
oh well, looks like more work than I'll get done this night with a cat sabotaging me...
ertes 2017-02-27 11:18:52
merijn: there, have fun: https://gist.github.com/esoeylemez/5ea02bc944af4b9fb9f963f06c9ed374
ertes 2017-02-27 11:18:53
=)
ertes 2017-02-27 11:19:34
System.Clock is from the 'clock' package, which is probably more expensive than criterion's clock
merijn 2017-02-27 11:20:16
ertes: Yeah, I got that much of your explanation. Fitting it into criterion is the hard part ;)
lyxia 2017-02-27 11:27:48
dolio: thanks for the help!
Carollshelby_ 2017-02-27 11:35:42
jezuz
Carollshelby_ 2017-02-27 11:35:47
look at all dem ppl
wizard1337 2017-02-27 11:35:49
given how important software is to the economy, why is there not an insane level of investment in programming language development
wizard1337 2017-02-27 11:35:57
?
merijn 2017-02-27 11:36:12
wizard1337: Same reason we're stuck with crappy OSes
merijn 2017-02-27 11:36:30
wizard1337: Business cares too much about backwards compatibility (of both code and their current employees)
wizard1337 2017-02-27 11:36:49
we have vms
merijn 2017-02-27 11:36:49
wizard1337: Also, long term investment in systems design has an unclear pay off/ROI, so they don't do it
wizard1337 2017-02-27 11:36:55
so thats solved
wizard1337 2017-02-27 11:37:09
you can always run windows xp in a vm ;)
wizard1337 2017-02-27 11:37:26
look at semtech,
wizard1337 2017-02-27 11:37:41
where semiconductor firms partnered to make a research org
wizard1337 2017-02-27 11:37:50
or we can do federal investment...
merijn 2017-02-27 11:39:03
Yeah...in theory
merijn 2017-02-27 11:39:22
In practice it's hard to get funding for systems research/engineering
Tuplanolla 2017-02-27 11:39:22
Every now and then some essential infrastructure breaks and only then do people realize it was all maintained by a single hobbyist from central Europe.
ertes 2017-02-27 11:39:40
also when people actually do invest in programming language development, you get stuff like go
merijn 2017-02-27 11:40:06
ertes: That's a bit disingenous
Tuplanolla 2017-02-27 11:40:07
The existence of programming languages is one of those things that most people take for granted.
merijn 2017-02-27 11:40:12
ertes: You also get Rust and Swift...
wizard1337 2017-02-27 11:40:17
lol central europe hobbyist
wizard1337 2017-02-27 11:40:43
C++ is such a joke
merijn 2017-02-27 11:41:05
wizard1337: You laugh, but wasn't it like GNU Octave that this month went like "welp, I can't pay my bills working on this, I need people to pay me", despite it being used by loads of companies to make big bucks...
wizard1337 2017-02-27 11:41:22
yeah but octave sucks
wizard1337 2017-02-27 11:41:30
and julia is replacing it
merijn 2017-02-27 11:41:33
Irrelevant
ertes 2017-02-27 11:42:01
merijn: both of those i could complain about, although rust at least seem to go into the right direction
merijn 2017-02-27 11:42:03
It's a language that's probably allowing millions of dollars of business to run and it's maintained by one guy who can't even live off it
ertes 2017-02-27 11:42:07
*seems
merijn 2017-02-27 11:42:19
ertes: Swift is a lot more in the right direction than Go or C# :p
merijn 2017-02-27 11:42:31
ertes: Apple could've made Objective-C# instead...
wizard1337 2017-02-27 11:43:34
visual objective C#++-- omega
wizard1337 2017-02-27 11:44:00
that will be my pl, anyone want to join, lol
