README.md (10936B)
1 <!-- 2 3 @license Apache-2.0 4 5 Copyright (c) 2018 The Stdlib Authors. 6 7 Licensed under the Apache License, Version 2.0 (the "License"); 8 you may not use this file except in compliance with the License. 9 You may obtain a copy of the License at 10 11 http://www.apache.org/licenses/LICENSE-2.0 12 13 Unless required by applicable law or agreed to in writing, software 14 distributed under the License is distributed on an "AS IS" BASIS, 15 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 See the License for the specific language governing permissions and 17 limitations under the License. 18 19 --> 20 21 # timeit 22 23 > Time a snippet. 24 25 <!-- Section to include introductory text. Make sure to keep an empty line after the intro `section` element and another before the `/section` close. --> 26 27 <section class="intro"> 28 29 </section> 30 31 <!-- /.intro --> 32 33 <!-- Package usage documentation. --> 34 35 <section class="usage"> 36 37 ## Usage 38 39 ```javascript 40 var timeit = require( '@stdlib/utils/timeit' ); 41 ``` 42 43 #### timeit( code, \[options,] clbk ) 44 45 Times a snippet. 46 47 ```javascript 48 var code = 'var x = Math.pow( Math.random(), 3 );'; 49 code += 'if ( x !== x ) {'; 50 code += 'throw new Error( \'Something went wrong.\' );'; 51 code += '}'; 52 53 timeit( code, done ); 54 55 function done( error, results ) { 56 if ( error ) { 57 throw error; 58 } 59 console.dir( results ); 60 /* e.g., => 61 { 62 'iterations': 1000000, 63 'repeats': 3, 64 'min': [0,135734733], // [seconds,nanoseconds] 65 'elapsed': 0.135734733, // seconds 66 'rate': 7367311.062526641, // iterations/second 67 'times': [ // raw timing results 68 [0,145641393], 69 [0,135734733], 70 [0,140462721] 71 ] 72 } 73 */ 74 } 75 ``` 76 77 The function supports the following `options`: 78 79 - **before**: setup code. Default: `""`. 80 - **after**: cleanup code. Default: `""`. 81 - **iterations**: number of iterations. If `null`, the number of iterations is determined by trying successive powers of `10` until the total time is at least `0.1` seconds. Default: `1e6`. 82 - **repeats**: number of repeats. Default: `3`. 83 - **asynchronous**: `boolean` indicating whether a snippet is asynchronous. Default: `false`. 84 85 To perform any setup or initialization, provide setup code. 86 87 ```javascript 88 var setup = 'var randu = require( \'@stdlib/random/base/randu\' );'; 89 setup += 'var pow = require( \'@stdlib/math/base/special/pow\' );'; 90 91 var code = 'var x = pow( randu(), 3 );'; 92 code += 'if ( x !== x ) {'; 93 code += 'throw new Error( \'Something went wrong.\' );'; 94 code += '}'; 95 96 var opts = { 97 'before': setup 98 }; 99 100 timeit( code, opts, done ); 101 102 function done( error, results ) { 103 if ( error ) { 104 throw error; 105 } 106 console.dir( results ); 107 } 108 ``` 109 110 To perform any cleanup, provide cleanup code. 111 112 ```javascript 113 var setup = 'var randu = require( \'@stdlib/random/base/randu\' );'; 114 setup += 'var hypot = require( \'@stdlib/math/base/special/hypot\' );'; 115 116 var code = 'var h = hypot( randu()*10, randu()*10 );'; 117 code += 'if ( h < 0 || h > 200 ) {'; 118 code += 'throw new Error( \'Something went wrong.\' );'; 119 code += '}'; 120 121 var cleanup = 'if ( h !== h ) {'; 122 cleanup += 'throw new Error( \'Something went wrong.\' );'; 123 cleanup += '}'; 124 125 var opts = { 126 'before': setup, 127 'after': cleanup 128 }; 129 130 timeit( code, opts, done ); 131 132 function done( error, results ) { 133 if ( error ) { 134 throw error; 135 } 136 console.dir( results ); 137 } 138 ``` 139 140 To time an asynchronous snippet, set the `asynchronous` option to `true`. 141 142 ```javascript 143 var code = 'var x = Math.pow( Math.random(), 3 );'; 144 code += 'if ( x !== x ) {'; 145 code += 'var err = new Error( \'Something went wrong.\' );'; 146 code += 'next( err );'; 147 code += '}'; 148 code += 'process.nextTick( next );'; 149 150 var opts = { 151 'iterations': 1e2, 152 'asynchronous': true 153 }; 154 155 timeit( code, opts, done ); 156 157 function done( error, results ) { 158 if ( error ) { 159 throw error; 160 } 161 console.dir( results ); 162 } 163 ``` 164 165 If `asynchronous` is `true`, the implementation assumes that `before`, `after`, and `code` snippets are **all** asynchronous. Accordingly, these snippets should invoke a `next( [error] )` callback once complete. For example, given the following snippet, 166 167 <!-- run-disable --> 168 169 <!-- eslint-disable --> 170 171 ```javascript 172 setTimeout( done, 0 ); 173 174 function done( error ) { 175 if ( error ) { 176 return next( error ); 177 } 178 next(); 179 } 180 ``` 181 182 the implementation wraps the snippet within a function having the following signature 183 184 ```javascript 185 function wrapped( state, next ) { 186 setTimeout( done, 0 ); 187 188 function done( error ) { 189 if ( error ) { 190 return next( error ); 191 } 192 next(); 193 } 194 } 195 ``` 196 197 The `state` parameter is simply an empty `{}` which allows the `before`, `after`, and `code` snippets to share state. 198 199 ```javascript 200 function before( state, next ) { 201 state.counter = 0; 202 } 203 204 function code( state, next ) { 205 setTimeout( done, 0 ); 206 207 function done( error ) { 208 if ( error ) { 209 return next( error ); 210 } 211 state.counter += 1; 212 next(); 213 } 214 } 215 216 function after( state, next ) { 217 var err; 218 if ( state.counter !== state.counter ) { 219 err = new Error( 'Something went wrong!' ); 220 return next( err ); 221 } 222 next(); 223 } 224 ``` 225 226 </section> 227 228 <!-- /.usage --> 229 230 <!-- Package usage notes. Make sure to keep an empty line after the `section` element and another before the `/section` close. --> 231 232 <section class="notes"> 233 234 ## Notes 235 236 - Snippets **always** run in [strict mode][strict-mode]. 237 - **Always** verify results. Doing so prevents the compiler from performing dead code elimination and other optimization techniques, which would render timing results meaningless. 238 - Executed code is **not** sandboxed and has access to the global state. You are **strongly** advised **against** timing untrusted code. To time untrusted code, do so in an isolated environment (e.g., a separate process with restricted access to both global state and the host environment). 239 - Wrapping asynchronous code **does** add overhead, but, in most cases, the overhead should be negligible compared to the execution cost of the timed snippet. 240 - Ensure that, when `asynchronous` is `true`, the main `code` snippet is actually asynchronous. If a snippet releases the [zalgo][zalgo], an error complaining about exceeding the maximum call stack size is highly likely. 241 - While many benchmark frameworks calculate various statistics over raw timing results (e.g., mean and standard deviation), do **not** do this. Instead, consider the fastest time an approximate lower bound for how fast an environment can execute a snippet. Slower times are more likely attributable to other processes interfering with timing accuracy rather than attributable to variability in JavaScript's speed. In which case, the minimum time is most likely the only result of interest. When considering all raw timing results, apply common sense rather than statistics. 242 243 </section> 244 245 <!-- /.notes --> 246 247 <!-- Package usage examples. --> 248 249 <section class="examples"> 250 251 ## Examples 252 253 <!-- eslint no-undef: "error" --> 254 255 ```javascript 256 var join = require( 'path' ).join; 257 var readFileSync = require( '@stdlib/fs/read-file' ).sync; 258 var timeit = require( '@stdlib/utils/timeit' ); 259 260 var before; 261 var code; 262 var opts; 263 264 before = readFileSync( join( __dirname, 'examples', 'before.txt' ), 'utf8' ); 265 code = readFileSync( join( __dirname, 'examples', 'code.txt' ), 'utf8' ); 266 267 opts = { 268 'iterations': 1e6, 269 'repeats': 5, 270 'before': before 271 }; 272 273 timeit( code, opts, done ); 274 275 function done( error, results ) { 276 if ( error ) { 277 throw error; 278 } 279 console.dir( results ); 280 } 281 ``` 282 283 </section> 284 285 <!-- /.examples --> 286 287 <!-- Section for describing a command-line interface. --> 288 289 * * * 290 291 <section class="cli"> 292 293 ## CLI 294 295 <!-- CLI usage documentation. --> 296 297 <section class="usage"> 298 299 ### Usage 300 301 ```text 302 Usage: timeit [options] [<code>] 303 304 Options: 305 306 -h, --help Print this message. 307 -V, --version Print the package version. 308 --iterations iter Number of iterations. 309 --repeats repeats Number of repeats. Default: 3. 310 --before setup Setup code. 311 --after cleanup Cleanup code. 312 --async Time asynchronous code. 313 --format fmt Output format: pretty, csv, json. Default: pretty. 314 ``` 315 316 </section> 317 318 <!-- /.usage --> 319 320 <!-- CLI usage notes. Make sure to keep an empty line after the `section` element and another before the `/section` close. --> 321 322 <section class="notes"> 323 324 ### Notes 325 326 - When the output format is `csv`, the output consists of **only** raw timing results. 327 - If not explicitly provided `--iterations`, the implementation tries successive powers of `10` until the total time is at least `0.1` seconds. 328 329 </section> 330 331 <!-- /.notes --> 332 333 <!-- CLI usage examples. --> 334 335 <section class="examples"> 336 337 ### Examples 338 339 ```bash 340 $ timeit "$(cat ./examples/code.txt)" --before "$(cat ./examples/before.txt)" --iterations 1000000 341 342 iterations: 1000000 343 repeats: 3 344 iterations/s: 7261975.851461222 345 elapsed time: 0.13770357 sec 346 lower bound: 0.13770357 usec/iteration 347 ``` 348 349 To output results as JSON, 350 351 ```bash 352 $ timeit "$(cat ./examples/code.txt)" --before "$(cat ./examples/before.txt)" --iterations 1000000 --format json 353 {"iterations":1000000,"repeats":3,"min":[0,132431806],"elapsed":0.132431806,"rate":7551056.1261997735,"times":[[0,142115140],[0,132431806],[0,134808376]]} 354 ``` 355 356 To output results as comma-separated values ([CSV][csv]), 357 358 ```bash 359 $ timeit "$(cat ./examples/code.txt)" --before "$(cat ./examples/before.txt)" --iterations 1000000 --format csv 360 seconds,nanoseconds 361 0,139365407 362 0,138033545 363 0,135175834 364 ``` 365 366 To use as part of a pipeline, 367 368 ```bash 369 $ cat ./examples/code.txt | timeit --before "$(cat ./examples/before.txt)" --iterations 1000000 370 371 iterations: 1000000 372 repeats: 3 373 iterations/s: 7433536.674260073 374 elapsed time: 0.134525468 sec 375 lower bound: 0.134525468 usec/iteration 376 ``` 377 378 </section> 379 380 <!-- /.examples --> 381 382 </section> 383 384 <!-- /.cli --> 385 386 <!-- Section to include cited references. If references are included, add a horizontal rule *before* the section. Make sure to keep an empty line after the `section` element and another before the `/section` close. --> 387 388 * * * 389 390 <section class="references"> 391 392 ## References 393 394 - Chen, Jiahao, and Jarrett Revels. 2016. "Robust benchmarking in noisy environments." _CoRR_ abs/1608.04295 (August). <http://arxiv.org/abs/1608.04295>. 395 396 </section> 397 398 <!-- /.references --> 399 400 <!-- Section for all links. Make sure to keep an empty line after the `section` element and another before the `/section` close. --> 401 402 <section class="links"> 403 404 [strict-mode]: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Strict_mode 405 406 [csv]: https://tools.ietf.org/html/rfc4180 407 408 [zalgo]: http://blog.izs.me/post/59142742143/designing-apis-for-asynchrony 409 410 </section> 411 412 <!-- /.links -->