class: ur-title, center, middle, title-slide # BST430 Lecture 13 ## Functions (ii) and Iteration ### Andrew McDavid ### U of Rochester ### 2021-10-23 (updated: 2021-10-27) --- ## Where were we? Where are we going? * Wrote an R function to compute the difference between the max and min of a numeric vector. * Validated the function's only argument * Informally, we verified that it worked OK. Now, we generalize this function, learn more technical details about R functions, and set default values for some arguments. --- ## Recall our function: ```r max_minus_min = function(x) { if(!is.numeric(x)) { stop('I am so sorry, but this function only works for numeric input!\n', 'You have provided an object of class: ', class(x)[1]) } max(x) - min(x) } ``` --- ## Generalize our function to other quantiles The max and the min are special cases of a __quantile__. Here are other special cases you may have heard of: * median = 0.5 quantile * 1st quartile = 0.25 quantile * 3rd quartile = 0.75 quantile In [box plots][wiki-boxplot], the rectangle typically runs from the 1st--3rd quartile, with a line at the median. If `\(q\)` is the `\(p\)`-th quantile of a set of `\(n\)` observations, approximately `\(pn\)` of the observations are less than `\(q\)` and `\((1 - p)n\)` are greater than `\(q\)`. Yeah, you need to worry about rounding to an integer and less/greater than or equal to... Let's generalize our function to take the difference between any two quantiles, min and max being **quantile 0** and **quantile 1** respectively. --- ## Get something that works, again The eventual inputs to our new function will be the data `x` and two probabilities. First, play around with the `quantile()` function. Convince yourself you know how to use it, for example, by cross-checking your results with other built-in functions. --- ```r library(gapminder) quantile(gapminder$lifeExp) ``` ``` ## 0% 25% 50% 75% 100% ## 23.5990 48.1980 60.7125 70.8455 82.6030 ``` ```r quantile(gapminder$lifeExp, probs = 0.5) ``` ``` ## 50% ## 60.7125 ``` ```r median(gapminder$lifeExp) ``` ``` ## [1] 60.7125 ``` ```r mean(gapminder$lifeExp < median(gapminder$lifeExp)) ``` ``` ## [1] 0.5 ``` --- Now write a code snippet that takes the difference between two quantiles. --- ## Turn the working interactive code into a function, again I'll use `qdiff` as the base of our function's name. I copy the overall structure from our previous "max minus min" work but replace the guts of the function with the more general code we just developed. ```r qdiff1 = function(x, probs) { stopifnot(is.numeric(x)) the_quantiles = quantile(x = x, probs = probs) max(the_quantiles) - min(the_quantiles) } ``` --- ```r qdiff1(gapminder$lifeExp, probs = c(0.25, 0.75)) ``` ``` ## [1] 22.6475 ``` ```r IQR(gapminder$lifeExp) # hey, we've reinvented IQR ``` ``` ## [1] 22.6475 ``` ```r qdiff1(gapminder$lifeExp, probs = c(0, 1)) ``` ``` ## [1] 59.004 ``` ```r max_minus_min(gapminder$lifeExp) ``` ``` ## [1] 59.004 ``` Again we do some informal tests against familiar results and external implementations. --- ## Argument names: freedom and conventions I want you to understand the importance of argument names. I can name my arguments almost anything I like. Proof: ```r qdiff2 = function(zeus, hera) { stopifnot(is.numeric(zeus)) the_quantiles = quantile(x = zeus, probs = hera) max(the_quantiles) - min(the_quantiles) } qdiff2(zeus = gapminder$lifeExp, hera = 0:1) ``` ``` ## [1] 59.004 ``` While I can name my arguments after Greek gods, it's usually a bad idea. Take all opportunities to make things more self-explanatory via meaningful names. --- ## Argument names: freedom and conventions * If you are going to pass along the arguments of your function to another built-in function, consider copying the argument names. * Unless you have a good reason to do your own thing (some argument names are bad!) * The similarity or equivalence of the formal names of the function arguments __accomplishes nothing__ as far as R is concerned; it is solely for the benefit of humans reading, writing, and using the code. * Which is very important! --- ## What a function returns By default, a function returns the result of the last line of the body. I am just letting that happen with the line `max(the_quantiles) - min(the_quantiles)`. However, there is an explicit function for this: `return()`. I could just as easily make this the last line of my function's body: ```r return(max(the_quantiles) - min(the_quantiles)) ``` You absolutely must use `return()` if you want to return early based on some condition, i.e. before execution gets to the last line of the body. Otherwise, you can decide your own conventions about when you use `return()` and when you don't -- I find it less bug-prone to explicitly return, and this is the course style guide. --- ## Default values: freedom to NOT specify the arguments What happens if we call our function but neglect to specify the probabilities? ```r qdiff1(gapminder$lifeExp) ``` ``` ## Error in quantile(x = x, probs = probs): argument "probs" is missing, with no default ``` Oops! At the moment, this causes a fatal error. It can be nice to provide some reasonable default values for certain arguments. In our case, it would be crazy to specify a default value for the primary input `x`, but very kind to specify a default for `probs`. --- ## Default values: freedom to NOT specify the arguments We started by focusing on the max and the min, so I think those make reasonable defaults. Here's how to specify that in a function definition. ```r qdiff3 = function(x, probs = c(0, 1)) { stopifnot(is.numeric(x)) the_quantiles = quantile(x, probs) max(the_quantiles) - min(the_quantiles) } ``` --- Again we check how the function works, in old examples and new, specifying the `probs` argument and not. ```r qdiff3(gapminder$lifeExp) ``` ``` ## [1] 59.004 ``` ```r max_minus_min(gapminder$lifeExp) ``` ``` ## [1] 59.004 ``` ```r qdiff3(gapminder$lifeExp, c(0.1, 0.9)) ``` ``` ## [1] 33.5862 ``` --- ## Check the validity of arguments, again __Exercise:__ upgrade our argument validity checks in light of the new argument `probs`. Application exercise -- browse [rstudio cloud](https://rstudio.cloud/spaces/162296/project/3099211) and complete part II. --- ## Wrap-up and what's next? Here's the function we've written so far: ```r qdiff3 ``` ``` ## function(x, probs = c(0, 1)) { ## stopifnot(is.numeric(x)) ## the_quantiles = quantile(x, probs) ## max(the_quantiles) - min(the_quantiles) ## } ## <bytecode: 0x7fb41446c618> ``` --- What we've accomplished: * We've generalized our first function to take a difference between arbitrary quantiles. * We've specified default values for the probabilities that set the quantiles. --- <!--Original content: https://stat545.com/block011_write-your-own-function-03.html--> ## Where were we? Where are we going? In this part, we tackle `NA`s, the special argument `...` and formal testing. Let's keep our previous function around as a baseline. ```r qdiff3 = function(x, probs = c(0, 1)) { stopifnot(is.numeric(x)) the_quantiles = quantile(x, probs) max(the_quantiles) - min(the_quantiles) } ``` --- class: code70 ## Be proactive about `NA`s The Gapminder data is a fairly kind data set. In real life, missing data will make your life a living hell. Let's see how `quantile()` handles `NA`s: ```r z = gapminder$lifeExp z[3] = NA quantile(z) ``` ``` ## Error in quantile.default(z): missing values and NaN's not allowed if 'na.rm' is FALSE ``` ```r quantile(z, na.rm = TRUE) ``` ``` ## 0% 25% 50% 75% 100% ## 23.599 48.228 60.765 70.846 82.603 ``` So `quantile()` simply will not operate in the presence of `NA`s unless `na.rm = TRUE`. How shall we modify our function? --- If we wanted to hardwire `na.rm = TRUE`, we could. Focus on our call to `quantile()` inside our function definition. ```r qdiff4 = function(x, probs = c(0, 1)) { stopifnot(is.numeric(x)) the_quantiles = quantile(x, probs, na.rm = TRUE) max(the_quantiles) - min(the_quantiles) } qdiff4(gapminder$lifeExp) ``` ``` ## [1] 59.004 ``` ```r qdiff4(z) ``` ``` ## [1] 59.004 ``` This works but it is dangerous to invert the default behavior of a well-known built-in function and to provide the user with no way to override this. --- We could add an `na.rm =` argument to our own function. We might even enforce our preferred default -- but at least we're giving the user a way to control the behavior around `NA`s. ```r qdiff5 = function(x, probs = c(0, 1), na.rm = TRUE) { stopifnot(is.numeric(x)) the_quantiles = quantile(x, probs, na.rm = na.rm) max(the_quantiles) - min(the_quantiles) } qdiff5(gapminder$lifeExp) ``` ``` ## [1] 59.004 ``` ```r qdiff5(z) ``` ``` ## [1] 59.004 ``` ```r qdiff5(z, na.rm = FALSE) ``` ``` ## Error in quantile.default(x, probs, na.rm = na.rm): missing values and NaN's not allowed if 'na.rm' is FALSE ``` --- ### You could have lived a long and happy life without knowing there are [at least 9 different algorithms][rdocs-quantile] for computing quantiles. The practical significance of `quatile(x, type =)` in the Gapminder data is virtually nonexistent. But thanks to [@wrathematics][twitter-wrathematics], here's a small example where we can detect a difference due to `type`. ```r set.seed(1234) z = rnorm(10) quantile(z, type = 1) ``` ``` ## 0% 25% 50% 75% 100% ## -2.3456977 -0.8900378 -0.5644520 0.4291247 1.0844412 ``` ```r quantile(z, type = 4) ``` ``` ## 0% 25% 50% 75% 100% ## -2.345698 -1.048552 -0.564452 0.353277 1.084441 ``` ??? TLDR: If a quantile is not unambiguously equal to an observed data point, you must somehow average two data points. You can weight this average different ways, depending on the rest of the data, and `type =` controls this. --- ## The useful but mysterious `...` argument But let's say we want flexibility to specify how the quantiles are computed, but don't want to clutter the function's interface with this. This calls for the special `...` ("ellipsis"). ```r qdiff6 = function(x, probs = c(0, 1), na.rm = TRUE, ...) { the_quantiles = quantile(x = x, probs = probs, na.rm = na.rm, ...) max(the_quantiles) - min(the_quantiles) } ``` --- Now we can call our function, requesting that quantiles be computed in different ways. ```r qdiff6(z, probs = c(0.25, 0.75), type = 1) ``` ``` ## [1] 1.319163 ``` ```r qdiff6(z, probs = c(0.25, 0.75), type = 4) ``` ``` ## [1] 1.401829 ``` Marvel at the fact that we have passed `type = 1` through to `quantile()` *even though it was not a formal argument of our own function*. --- ### Notes on `...` In short, `...` is a catch-all, passing arbitrary arguments down to another function. You will encounter `...` in many built-in functions: see [`c()`][rdocs-c] or [`list()`][rdocs-list]. There are also downsides to `...`: - In a package, it's harder to create informative documentation for your user. - The quiet, absorbent properties of `...` mean it can silently swallow other named arguments, when the user has a typo in the name. - The [ellipsis package](https://ellipsis.r-lib.org) provides tools that help package developers use `...` more safely. --- ### Functional scope In R and in many languages, variables defined inside the function are local (private) to the function. ```r a = 10 my_function = function(a, x){ a = 5 a*x } ``` ```r my_function(1, 2) ``` ``` ## [1] 10 ``` ```r print(a) ``` ``` ## [1] 10 ``` --- ```r my_function(a, 2) ``` ``` ## [1] 10 ``` ```r print(a) ``` ``` ## [1] 10 ``` --- ### Unbound variables are searched for in the enclosing environments However, R uses something known as **lexical scope** to resolve conflicting variable names, and functions define what is technically called an enclosure.¹ ```r a = -5 my_function2 = function(x){ a*x } my_function2(2) ``` ``` ## [1] -10 ``` R looks at the enclosing environments of a function (what was being executed when it was defined) to resolve a binding. .footnote[[1] Here, and its S3 object orientation system, R often seems rather bizarre. It will make more sense if you consume your preferred toxicant, and read up about LISP in the 1980s. Perhaps.] --- ### This can lead to very frustating errors ```r data = dplyr::starwars process_data = function(dta){ region_summary = filter(dta, !is.na(result)) %>% group_by(region) %>% summarize(mean = mean(result)) # do some other stuff * left_join(data, region_summary) #Uhoh! This should have been dta! } ``` I have lost an entire days to this. --- ### Current environment is preferred. ```r a = -5 my_function3 = function(x){ a = 1e9 a*x } my_function3(2) ``` ``` ## [1] 2e+09 ``` ```r print(a) ``` ``` ## [1] -5 ``` --- ### Functions are first-class citizens in R A function is another type of object -- this means you can pass a function as an argument to a function! ```r quartic_eq = function(x) 3*x^4 - 10*x^3 - 20*x^2 + 10*x - 5 class(quartic_eq) ``` ``` ## [1] "function" ``` ```r body(quartic_eq) ``` ``` ## 3 * x^4 - 10 * x^3 - 20 * x^2 + 10 * x - 5 ``` ```r formals(quartic_eq) ``` ``` ## $x ``` Can also display a function's source by typing its name into the Console, or hovering over it and hitting F2. --- ```r curve(quartic_eq, from = -5, to = 5) ``` <img src="l13-functions-ii_files/figure-html/unnamed-chunk-26-1.png" width="60%" style="display: block; margin: auto;" /> ```r optimize(quartic_eq, interval = c(-5, 5)) ``` ``` ## $minimum ## [1] 3.406671 ## ## $objective ## [1] -194.343 ``` --- class: middle .hand[Iteration] --- ## Flow control: iteration It's a testament to the tidyverse that we have made it this far into the semester using purely `group_by` or vectorization¹ to accomplish our iteration -- applying an function to varying subsets of "the data." But sometimes you just gotta loop. .footnote[[1] If you don't know what that means, I have done my job 😸.] --- .pull-left[ ### Ways to explicitly iterate * `for`- iterate a **code block** over a vector * `while` - iterate a **code block** until a boolean is TRUE ] .pull-right[ ### Ways to implicitly iterate * `lapply` - iterate a **function** over a vector * `purrr` - iterate a **function** over a vector, but carefully * `apply` - iterate a **function** over an array * vectorized functions * `group_by` * recursion ] --- ### `for()` A `for()` loop increments a **counter** variable along a vector. It repeatedly runs a code block, called the **body** of the loop, with the counter set at its current value, until it runs through the vector ``` for( <COUNTER> in <VECTOR> ){ <BODY> } ``` where `<BODY>` contains a sequence of R expressions, as well as the keywords `break` or `continue`. --- ```r n = 5 log.vec = 0 for (i in seq_len(n)) { log.vec[i] = log(i) } log.vec ``` ``` ## [1] 0.0000000 0.6931472 1.0986123 1.3862944 1.6094379 ``` Here `i` is the counter and the vector we are iterating over is `1:n`. The body is the code in between the braces. --- ### `1:n` considered harmful Note that rather than using `1:n`, which when `n = 0` equals `1 0` we use `seq_len`, which protects us when the index isn't guaranteed to be positive: .pull-left[ ```r x = sort(runif(5)) n_negative = sum(x<0) for (i in 1:n_negative){ print(x[i]) } ``` ``` ## [1] 0.03999592 ## numeric(0) ``` ```r cat("Exited loop.") ``` ``` ## Exited loop. ``` ] .pull-right[ ```r x = sort(runif(5)) n_negative = sum(x<0) for (i in seq_len(n_negative)){ print(x[i]) } cat("Exited loop.") ``` ``` ## Exited loop. ``` ] Also see `seq_along`. --- class: code70 ### Breaking from the loop We can **break** out of a `for()` loop early (before the counter has been iterated over the whole vector), using `break` ```r n = 5 log.vec = 0 for (i in 1:n) { if (log(i) > 1) { cat("I'm outta here. I don't like numbers bigger than 1\n") break } log.vec[i] = log(i) } ``` ``` ## I'm outta here. I don't like numbers bigger than 1 ``` ```r log.vec ``` ``` ## [1] 0.0000000 0.6931472 ``` `continue` skips the remainder of the body of a `for()` loop, returning to the top of the body and incrementing the counter. --- ### Variations on standard `for()` loops Many different variations on standard `for()` are possible. Two common ones: - Nonnumeric counters: counter variable always gets iterated over a vector, but it doesn't have to be numeric - Nested loops: body of the `for()` loop can contain another `for()` loop (or several others) ```r for (str in c("Tatoosh", "Infimum", "McGrindleCat")) { cat(glue::glue("Free (OBO): {str}, one gently used cat.")) } ``` ``` ## Free (OBO): Tatoosh, one gently used cat.Free (OBO): Infimum, one gently used cat.Free (OBO): McGrindleCat, one gently used cat. ``` .footnote[[1] Just kidding, Tatoosh, we love you.] --- ### Nested `for()` loops -- I guess we're working with pairs ```r for (i in seq_len(4)) { for (j in i:4) { cat(paste(j,"")) } cat("\n") } ``` ``` ## 1 2 3 4 ## 2 3 4 ## 3 4 ## 4 ``` --- ### `while()` A `while()` loop repeatedly runs a code block, again called the **body**, until some condition is no longer true. --- ### Roll your own linear regression ```r X = model.matrix(~ scale(mass), data = dplyr::starwars) y = filter(starwars, !is.na(height) & !is.na(mass)) %>% pull(height) beta = matrix(0, nrow = ncol(X)) beta_new = matrix(1, nrow = ncol(X)) epsilon = 1e-4 i = 0 while (mean((beta - beta_new)^2) > epsilon) { print(glue::glue("At i = {i}, beta = ({beta[1]}, {beta[2]})")) beta = beta_new beta_new = beta + 0.02 * crossprod(X, y - X%*%beta) i = i + 1 } ``` ``` ## At i = 0, beta = (0, 0) ## At i = 1, beta = (1, 1) ## At i = 2, beta = (205.16, 5.35781787904754) ## At i = 3, beta = (168.4112, 4.66056701839993) ## At i = 4, beta = (175.025984, 4.77212715610355) ## At i = 5, beta = (173.83532288, 4.75427753407097) ## At i = 6, beta = (174.0496418816, 4.75713347359618) ``` --- ### Compare to `lm` ```r lm(height ~ scale(mass), data = dplyr::starwars) ``` ``` ## ## Call: ## lm(formula = height ~ scale(mass), data = dplyr::starwars) ## ## Coefficients: ## (Intercept) scale(mass) ## 174.017 4.757 ``` --- ### `for()` versus `while()` - `for()` is better when the number of times to repeat (values to iterate over) is clear in advance - `while()` is better when you can recognize when to stop once you're there, even if you can't guess it to begin with - `while()` is more general, in that every `for()` could be replaced with a `while()` (but not vice versa) - Can also use `while(TRUE)`. Repeat the body indefinitely, until something causes the flow to break (like `break`!). Useful if you don't want to check the termination condition until you've run `<BODY>` at least once. All of these require bookkeeping about the output (its length, how its indices map to the input data, etc). When possible use implicit iteration, which handles the bookkeeping for you. --- ## Implicit iteration `lapply`, `purrr::map_*`, and `apply` all iterate a function over slices of data. `lapply` and `purrr::map_*` work over vectors (both atomic and list), while `apply` works over matrices / arrays. Sometimes, you might combine `lapply` and friends with `split`, which chops a vector or data frame into a list of vectors by some discrete factor. However, you can often get the desired effect with a `group_by` + `summarize`. --- ### lapply .pull-left[ ```r weird_list = list(rnorm(5), stringr::fruit, TRUE, starwars) lapply(weird_list, length) ``` ``` ## [[1]] ## [1] 5 ## ## [[2]] ## [1] 80 ## ## [[3]] ## [1] 1 ## ## [[4]] ## [1] 14 ``` ] .pull-right[ ```r lapply(weird_list, class) ``` ``` ## [[1]] ## [1] "numeric" ## ## [[2]] ## [1] "character" ## ## [[3]] ## [1] "logical" ## ## [[4]] ## [1] "tbl_df" "tbl" "data.frame" ``` ] --- ### purrr::map_dfr This applies a function, which must return a data frame, to each element of a list, and binds the rows together. I reach for this often. ```r purrr::map_dfr(weird_list, function(x){ tibble(length = length(x), class = class(x)[1]) }, .id = 'item') ``` ``` ## # A tibble: 4 x 3 ## item length class ## <chr> <int> <chr> ## 1 1 5 numeric ## 2 2 80 character ## 3 3 1 logical ## 4 4 14 tbl_df ``` This uses an **anonymous** function -- defined but not named. In purrr, you can also create anonymous functions using [`~ .x`](https://purrr.tidyverse.org/articles/other-langs.html) syntax to save a few keystrokes. I mostly prefer to be explicit, because it makes it easier to debug. --- ### apply ```r (A = matrix(1:18, nrow = 3)) ``` ``` ## [,1] [,2] [,3] [,4] [,5] [,6] ## [1,] 1 4 7 10 13 16 ## [2,] 2 5 8 11 14 17 ## [3,] 3 6 9 12 15 18 ``` ```r count_odd = function(x) sum(x %% 2 > 0) apply(A, 1, count_odd) ``` ``` ## [1] 3 3 3 ``` ```r apply(A, 2, count_odd) ``` ``` ## [1] 2 1 2 1 2 1 ``` --- ### Recursion Recursion and (tail) iteration, in theory and in some languages, are equivalent¹. R is a bit fussy about recursion, however, and you can overflow the stack if you aren't careful. It's also less performant. The Collatz function is the recursion $$ f(n) = `\begin{cases} \text{stop} & n = 1 \\ n/2 & n \equiv 0 (\mod 2), \\ 3n + 1 & n \equiv 1 (\mod 2). \end{cases}` $$ and `\(a_i = f(a_{i-1})\)`. .footnote[[1] Though recursion makes my brain hurt more.] --- ### Collatz recursion ```r collatz = function(n){ cat(n) if(n == 1) return(invisible(1)) cat("->") if(n %% 2 == 1){ # odd return(collatz(3*n + 1)) } else{ # even return(collatz(n /2)) } } ``` --- ```r collatz(10) ``` ``` ## 10->5->16->8->4->2->1 ``` ```r collatz(101) ``` ``` ## 101->304->152->76->38->19->58->29->88->44->22->11->34->17->52->26->13->40->20->10->5->16->8->4->2->1 ``` ```r collatz(837799) ``` ``` ## 837799->2513398->1256699->3770098->1885049->5655148->2827574->1413787->4241362->2120681->6362044->3181022->1590511->4771534->2385767->7157302->3578651->10735954->5367977->16103932->8051966->4025983->12077950->6038975->18116926->9058463->27175390->13587695->40763086->20381543->61144630->30572315->91716946->45858473->137575420->68787710->34393855->103181566->51590783->154772350->77386175->232158526->116079263->348237790->174118895->522356686->261178343->783535030->391767515->1175302546->587651273->1762953820->881476910->440738455->1322215366->661107683->1983323050->991661525->2974984576->1487492288->743746144->371873072->185936536->92968268->46484134->23242067->69726202->34863101->104589304->52294652->26147326->13073663->39220990->19610495->58831486->29415743->88247230->44123615->132370846->66185423->198556270->99278135->297834406->148917203->446751610->223375805->670127416->335063708->167531854->83765927->251297782->125648891->376946674->188473337->565420012->282710006->141355003->424065010->212032505->636097516->318048758->159024379->477073138->238536569->715609708->357804854->178902427->536707282->268353641->805060924->402530462->201265231->603795694->301897847->905693542->452846771->1358540314->679270157->2037810472->1018905236->509452618->254726309->764178928->382089464->191044732->95522366->47761183->143283550->71641775->214925326->107462663->322387990->161193995->483581986->241790993->725372980->362686490->181343245->544029736->272014868->136007434->68003717->204011152->102005576->51002788->25501394->12750697->38252092->19126046->9563023->28689070->14344535->43033606->21516803->64550410->32275205->96825616->48412808->24206404->12103202->6051601->18154804->9077402->4538701->13616104->6808052->3404026->1702013->5106040->2553020->1276510->638255->1914766->957383->2872150->1436075->4308226->2154113->6462340->3231170->1615585->4846756->2423378->1211689->3635068->1817534->908767->2726302->1363151->4089454->2044727->6134182->3067091->9201274->4600637->13801912->6900956->3450478->1725239->5175718->2587859->7763578->3881789->11645368->5822684->2911342->1455671->4367014->2183507->6550522->3275261->9825784->4912892->2456446->1228223->3684670->1842335->5527006->2763503->8290510->4145255->12435766->6217883->18653650->9326825->27980476->13990238->6995119->20985358->10492679->31478038->15739019->47217058->23608529->70825588->35412794->17706397->53119192->26559596->13279798->6639899->19919698->9959849->29879548->14939774->7469887->22409662->11204831->33614494->16807247->50421742->25210871->75632614->37816307->113448922->56724461->170173384->85086692->42543346->21271673->63815020->31907510->15953755->47861266->23930633->71791900->35895950->17947975->53843926->26921963->80765890->40382945->121148836->60574418->30287209->90861628->45430814->22715407->68146222->34073111->102219334->51109667->153329002->76664501->229993504->114996752->57498376->28749188->14374594->7187297->21561892->10780946->5390473->16171420->8085710->4042855->12128566->6064283->18192850->9096425->27289276->13644638->6822319->20466958->10233479->30700438->15350219->46050658->23025329->69075988->34537994->17268997->51806992->25903496->12951748->6475874->3237937->9713812->4856906->2428453->7285360->3642680->1821340->910670->455335->1366006->683003->2049010->1024505->3073516->1536758->768379->2305138->1152569->3457708->1728854->864427->2593282->1296641->3889924->1944962->972481->2917444->1458722->729361->2188084->1094042->547021->1641064->820532->410266->205133->615400->307700->153850->76925->230776->115388->57694->28847->86542->43271->129814->64907->194722->97361->292084->146042->73021->219064->109532->54766->27383->82150->41075->123226->61613->184840->92420->46210->23105->69316->34658->17329->51988->25994->12997->38992->19496->9748->4874->2437->7312->3656->1828->914->457->1372->686->343->1030->515->1546->773->2320->1160->580->290->145->436->218->109->328->164->82->41->124->62->31->94->47->142->71->214->107->322->161->484->242->121->364->182->91->274->137->412->206->103->310->155->466->233->700->350->175->526->263->790->395->1186->593->1780->890->445->1336->668->334->167->502->251->754->377->1132->566->283->850->425->1276->638->319->958->479->1438->719->2158->1079->3238->1619->4858->2429->7288->3644->1822->911->2734->1367->4102->2051->6154->3077->9232->4616->2308->1154->577->1732->866->433->1300->650->325->976->488->244->122->61->184->92->46->23->70->35->106->53->160->80->40->20->10->5->16->8->4->2->1 ``` --- class: middle .hand[Formal unit tests] --- ## Use testthat for formal unit tests Until now, we've relied on informal tests of our evolving function. If you are going to use a function a lot, or you keep finding bugs using informal tests, you'll save a lot of time an effort with formal unit tests. My rules of thumb: * Once I have found the second bug, it's time to write the unit tests. It will be much quicker to fix the second bug, and it will provide me with assurance that I might actually find the third bug. * No bug exists until you have the failing unit test isolating it. The [testthat][testthat-web] package provides excellent facilities for this, with a distinct emphasis on automated unit testing of entire packages. However, it works even with our one measly function. --- We will construct a test with `test_that()` and, within it, we put one or more *expectations* that check actual against expected results. You can simply your informal, interactive tests into formal unit tests. Here are some examples of tests and indicative expectations. ```r library(testthat) test_that('invalid args are detected', { expect_error(qdiff6("eggplants are purple")) expect_error(qdiff6(iris)) }) ``` ``` ## Test passed 😸 ``` ```r test_that('NA handling works', { expect_error(qdiff6(c(1:5, NA), na.rm = FALSE)) expect_equal(qdiff6(c(1:5, NA)), 4) }) ``` ``` ## Test passed 🌈 ``` No news is good news! --- Let's see what test failure would look like. Let's revert to a version of our function that does no `NA` handling, then test for proper `NA` handling. We can watch it fail. ```r qdiff_no_NA = function(x, probs = c(0, 1)) { the_quantiles = quantile(x = x, probs = probs) max(the_quantiles) - min(the_quantiles) } test_that('NA handling works', { expect_that(qdiff_no_NA(c(1:5, NA)), equals(4)) }) ``` ``` ## ── Error (<text>:6:3): NA handling works ─────────────────────────────────────── ## Error: missing values and NaN's not allowed if 'na.rm' is FALSE ## Backtrace: ## 1. testthat::expect_that(qdiff_no_NA(c(1:5, NA)), equals(4)) ## 6. global::qdiff_no_NA(c(1:5, NA)) ## 8. stats:::quantile.default(x = x, probs = probs) ``` --- ## Resources Hadley Wickham's book [Advanced R][adv-r] [-@wickham2015a]: * Section on [function arguments][adv-r-fxn-args] * Section on [return values][adv-r-return-values] * Tidyverse principles guide provides further guidance on the design of functions that take `...` in [Data, dots, details](https://principles.tidyverse.org/dots-position.html). --- # Acknowledgments Adapted from Jenny Bryan's STAT545: https://stat545.com/functions-part2.html https://stat545.com/functions-part3.html And from Ryan Tibshirani's Statistics 36-350: https://www.stat.cmu.edu/~ryantibs/statcomp/lectures/iteration.html [cran]: https://cloud.r-project.org [cran-faq]: https://cran.r-project.org/faqs.html [cran-R-admin]: http://cran.r-project.org/doc/manuals/R-admin.html [cran-add-ons]: https://cran.r-project.org/doc/manuals/R-admin.html#Add_002don-packages [r-proj]: https://www.r-project.org [stat-545]: https://stat545.com [software-carpentry]: https://software-carpentry.org [cran-r-extensions]: https://cran.r-project.org/doc/manuals/r-release/R-exts.html <!--RStudio Links--> [rstudio-preview]: https://www.rstudio.com/products/rstudio/download/preview/ [rstudio-official]: https://www.rstudio.com/products/rstudio/#Desktop [rstudio-workbench]: https://www.rstudio.com/wp-content/uploads/2014/04/rstudio-workbench.png [rstudio-support]: https://support.rstudio.com/hc/en-us [rstudio-R-help]: https://support.rstudio.com/hc/en-us/articles/200552336-Getting-Help-with-R [rstudio-customizing]: https://support.rstudio.com/hc/en-us/articles/200549016-Customizing-RStudio [rstudio-key-shortcuts]: https://support.rstudio.com/hc/en-us/articles/200711853-Keyboard-Shortcuts [rstudio-command-history]: https://support.rstudio.com/hc/en-us/articles/200526217-Command-History [rstudio-using-projects]: https://support.rstudio.com/hc/en-us/articles/200526207-Using-Projects [rstudio-code-snippets]: https://support.rstudio.com/hc/en-us/articles/204463668-Code-Snippets [rstudio-dplyr-cheatsheet-download]: https://github.com/rstudio/cheatsheets/raw/master/data-transformation.pdf [rstudio-regex-cheatsheet]: https://www.rstudio.com/wp-content/uploads/2016/09/RegExCheatsheet.pdf [rstudio-devtools]: https://www.rstudio.com/products/rpackages/devtools/ <!--HappyGitWithR Links--> [happy-git]: https://happygitwithr.com [hg-install-git]: https://happygitwithr.com/install-git.html [hg-git-client]: https://happygitwithr.com/git-client.html [hg-github-account]: https://happygitwithr.com/github-acct.html [hg-install-r-rstudio]: https://happygitwithr.com/install-r-rstudio.html [hg-connect-intro]: https://happygitwithr.com/connect-intro.html [hg-browsability]: https://happygitwithr.com/workflows-browsability.html [hg-shell]: https://happygitwithr.com/shell.html <!--Package Links--> [rmarkdown]: https://rmarkdown.rstudio.com [knitr-faq]: https://yihui.name/knitr/faq/ [tidyverse-main-page]: https://www.tidyverse.org [tidyverse-web]: https://tidyverse.tidyverse.org [tidyverse-github]: https://github.com/hadley/tidyverse [dplyr-web]: https://dplyr.tidyverse.org [dplyr-cran]: https://CRAN.R-project.org/package=dplyr [dplyr-github]: https://github.com/hadley/dplyr [dplyr-vignette-intro]: https://cran.r-project.org/web/packages/dplyr/vignettes/dplyr.html [dplyr-vignette-window-fxns]: https://cran.r-project.org/web/packages/dplyr/vignettes/window-functions.html [dplyr-vignette-two-table]: https://dplyr.tidyverse.org/articles/two-table.html [lubridate-web]: https://lubridate.tidyverse.org [lubridate-cran]: https://CRAN.R-project.org/package=lubridate [lubridate-github]: https://github.com/tidyverse/lubridate [lubridate-vignette]: https://cran.r-project.org/web/packages/lubridate/vignettes/lubridate.html [tidyr-web]: https://tidyr.tidyverse.org [tidyr-cran]: https://CRAN.R-project.org/package=tidyr [readr-web]: https://readr.tidyverse.org [readr-vignette-intro]: https://cran.r-project.org/web/packages/readr/vignettes/readr.html [stringr-web]: https://stringr.tidyverse.org [stringr-cran]: https://CRAN.R-project.org/package=stringr [ggplot2-web]: https://ggplot2.tidyverse.org [ggplot2-tutorial]: https://github.com/jennybc/ggplot2-tutorial [ggplot2-reference]: https://docs.ggplot2.org/current/ [ggplot2-cran]: https://CRAN.R-project.org/package=ggplot2 [ggplot2-github]: https://github.com/tidyverse/ggplot2 [ggplot2-theme-args]: https://ggplot2.tidyverse.org/reference/ggtheme.html#arguments [gapminder-web]: https://www.gapminder.org [gapminder-cran]: https://CRAN.R-project.org/package=gapminder [assertthat-cran]: https://CRAN.R-project.org/package=assertthat [assertthat-github]: https://github.com/hadley/assertthat [ensurer-cran]: https://CRAN.R-project.org/package=ensurer [ensurer-github]: https://github.com/smbache/ensurer [assertr-cran]: https://CRAN.R-project.org/package=assertr [assertr-github]: https://github.com/ropensci/assertr [assertive-cran]: https://CRAN.R-project.org/package=assertive [assertive-bitbucket]: https://bitbucket.org/richierocks/assertive/src/master/ [testthat-cran]: https://CRAN.R-project.org/package=testthat [testthat-github]: https://github.com/r-lib/testthat [testthat-web]: https://testthat.r-lib.org [viridis-cran]: https://CRAN.R-project.org/package=viridis [viridis-github]: https://github.com/sjmgarnier/viridis [viridis-vignette]: https://cran.r-project.org/web/packages/viridis/vignettes/intro-to-viridis.html [colorspace-cran]: https://CRAN.R-project.org/package=colorspace [colorspace-vignette]: https://cran.r-project.org/web/packages/colorspace/vignettes/hcl-colors.pdf [cowplot-cran]: https://CRAN.R-project.org/package=cowplot [cowplot-github]: https://github.com/wilkelab/cowplot [cowplot-vignette]: https://cran.r-project.org/web/packages/cowplot/vignettes/introduction.html [devtools-cran]: https://CRAN.R-project.org/package=devtools [devtools-github]: https://github.com/r-lib/devtools [devtools-web]: https://devtools.r-lib.org [devtools-cheatsheet]: https://www.rstudio.com/wp-content/uploads/2015/03/devtools-cheatsheet.pdf [devtools-cheatsheet-old]: https://rawgit.com/rstudio/cheatsheets/master/package-development.pdf [devtools-1-6]: https://blog.rstudio.com/2014/10/02/devtools-1-6/ [devtools-1-8]: https://blog.rstudio.com/2015/05/11/devtools-1-9-0/ [devtools-1-9-1]: https://blog.rstudio.com/2015/09/13/devtools-1-9-1/ [googlesheets-cran]: https://CRAN.R-project.org/package=googlesheets [googlesheets-github]: https://github.com/jennybc/googlesheets [tidycensus-cran]: https://CRAN.R-project.org/package=tidycensus [tidycensus-github]: https://github.com/walkerke/tidycensus [tidycensus-web]: https://walkerke.github.io/tidycensus/index.html [fs-web]: https://fs.r-lib.org/index.html [fs-cran]: https://CRAN.R-project.org/package=fs [fs-github]: https://github.com/r-lib/fs [plumber-web]: https://www.rplumber.io [plumber-docs]: https://www.rplumber.io/docs/ [plumber-github]: https://github.com/trestletech/plumber [plumber-cran]: https://CRAN.R-project.org/package=plumber [plyr-web]: http://plyr.had.co.nz [magrittr-web]: https://magrittr.tidyverse.org [forcats-web]: https://forcats.tidyverse.org [glue-web]: https://glue.tidyverse.org [stringi-cran]: https://CRAN.R-project.org/package=stringi [rex-github]: https://github.com/kevinushey/rex [rcolorbrewer-cran]: https://CRAN.R-project.org/package=RColorBrewer [dichromat-cran]: https://CRAN.R-project.org/package=dichromat [rdryad-web]: https://docs.ropensci.org/rdryad/ [rdryad-cran]: https://CRAN.R-project.org/package=rdryad [rdryad-github]: https://github.com/ropensci/rdryad [roxygen2-cran]: https://CRAN.R-project.org/package=roxygen2 [roxygen2-vignette]: https://cran.r-project.org/web/packages/roxygen2/vignettes/rd.html [shinythemes-web]: https://rstudio.github.io/shinythemes/ [shinythemes-cran]: https://CRAN.R-project.org/package=shinythemes [shinyjs-web]: https://deanattali.com/shinyjs/ [shinyjs-cran]: https://CRAN.R-project.org/package=shinyjs [shinyjs-github]: https://github.com/daattali/shinyjs [leaflet-web]: https://rstudio.github.io/leaflet/ [leaflet-cran]: https://CRAN.R-project.org/package=leaflet [leaflet-github]: https://github.com/rstudio/leaflet [ggvis-web]: https://ggvis.rstudio.com [ggvis-cran]: https://CRAN.R-project.org/package=ggvis [usethis-web]: https://usethis.r-lib.org [usethis-cran]: https://CRAN.R-project.org/package=usethis [usethis-github]: https://github.com/r-lib/usethis [pkgdown-web]: https://pkgdown.r-lib.org [gh-github]: https://github.com/r-lib/gh [httr-web]: https://httr.r-lib.org [httr-cran]: https://CRAN.R-project.org/package=httr [httr-github]: https://github.com/r-lib/httr [gistr-web]: https://docs.ropensci.org/gistr [gistr-cran]: https://CRAN.R-project.org/package=gistr [gistr-github]: https://github.com/ropensci/gistr [rvest-web]: https://rvest.tidyverse.org [rvest-cran]: https://CRAN.R-project.org/package=rvest [rvest-github]: https://github.com/tidyverse/rvest [xml2-web]: https://xml2.r-lib.org [xml2-cran]: https://CRAN.R-project.org/package=xml2 [xml2-github]: https://github.com/r-lib/xml2 [jsonlite-paper]: https://arxiv.org/abs/1403.2805 [jsonlite-cran]: https://CRAN.R-project.org/package=jsonlite [jsonlite-github]: https://github.com/jeroen/jsonlite [readxl-web]: https://readxl.tidyverse.org [readxl-github]: https://github.com/tidyverse/readxl [readxl-cran]: https://CRAN.R-project.org/package=readxl [janitor-web]: http://sfirke.github.io/janitor/ [janitor-cran]: https://CRAN.R-project.org/package=janitor [janitor-github]: https://github.com/sfirke/janitor [purrr-web]: https://purrr.tidyverse.org [curl-cran]: https://CRAN.R-project.org/package=curl <!--Shiny links--> [shinydashboard-web]: https://rstudio.github.io/shinydashboard/ [shinydashboard-cran]: https://CRAN.R-project.org/package=shinydashboard [shinydashboard-github]: https://github.com/rstudio/shinydashboard [shiny-official-web]: https://shiny.rstudio.com [shiny-official-tutorial]: https://shiny.rstudio.com/tutorial/ [shiny-cheatsheet]: https://shiny.rstudio.com/images/shiny-cheatsheet.pdf [shiny-articles]: https://shiny.rstudio.com/articles/ [shiny-bookdown]: https://bookdown.org/yihui/rmarkdown/shiny-documents.html [shiny-google-groups]: https://groups.google.com/forum/#!forum/shiny-discuss [shiny-stack-overflow]: https://stackoverflow.com/questions/tagged/shiny [shinyapps-web]: https://www.shinyapps.io [shiny-server-setup]: https://deanattali.com/2015/05/09/setup-rstudio-shiny-server-digital-ocean/ [shiny-reactivity]: https://shiny.rstudio.com/articles/understanding-reactivity.html [shiny-debugging]: https://shiny.rstudio.com/articles/debugging.html [shiny-server]: https://www.rstudio.com/products/shiny/shiny-server/ <!--Publications--> [adv-r]: http://adv-r.had.co.nz [adv-r-fxns]: http://adv-r.had.co.nz/Functions.html [adv-r-dsl]: http://adv-r.had.co.nz/dsl.html [adv-r-defensive-programming]: http://adv-r.had.co.nz/Exceptions-Debugging.html#defensive-programming [adv-r-fxn-args]: http://adv-r.had.co.nz/Functions.html#function-arguments [adv-r-return-values]: http://adv-r.had.co.nz/Functions.html#return-values [adv-r-closures]: http://adv-r.had.co.nz/Functional-programming.html#closures [r4ds]: https://r4ds.had.co.nz [r4ds-transform]: https://r4ds.had.co.nz/transform.html [r4ds-strings]: https://r4ds.had.co.nz/strings.html [r4ds-readr-strings]: https://r4ds.had.co.nz/data-import.html#readr-strings [r4ds-dates-times]: https://r4ds.had.co.nz/dates-and-times.html [r4ds-data-import]: http://r4ds.had.co.nz/data-import.html [r4ds-relational-data]: https://r4ds.had.co.nz/relational-data.html [r4ds-pepper-shaker]: https://r4ds.had.co.nz/vectors.html#lists-of-condiments [r-pkgs2]: https://r-pkgs.org/index.html [r-pkgs2-whole-game]: https://r-pkgs.org/whole-game.html [r-pkgs2-description]: https://r-pkgs.org/description.html [r-pkgs2-man]: https://r-pkgs.org/man.htm [r-pkgs2-tests]: https://r-pkgs.org/tests.html [r-pkgs2-namespace]: https://r-pkgs.org/namespace.html [r-pkgs2-vignettes]: https://r-pkgs.org/vignettes.html [r-pkgs2-release]: https://r-pkgs.org/release.html [r-pkgs2-r-code]: https://r-pkgs.org/r.html#r [r-graphics-cookbook]: http://shop.oreilly.com/product/0636920023135.do [cookbook-for-r]: http://www.cookbook-r.com [cookbook-for-r-graphs]: http://www.cookbook-r.com/Graphs/ [cookbook-for-r-multigraphs]: http://www.cookbook-r.com/Graphs/Multiple_graphs_on_one_page_(ggplot2)/ [elegant-graphics-springer]: https://www.springer.com/gp/book/9780387981413 [testthat-article]: https://journal.r-project.org/archive/2011-1/RJournal_2011-1_Wickham.pdf [worry-about-color]: https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=2&cad=rja&uact=8&ved=2ahUKEwi0xYqJ8JbjAhWNvp4KHViYDxsQFjABegQIABAC&url=https%3A%2F%2Fwww.researchgate.net%2Fprofile%2FAhmed_Elhattab2%2Fpost%2FPlease_suggest_some_good_3D_plot_tool_Software_for_surface_plot%2Fattachment%2F5c05ba35cfe4a7645506948e%2FAS%253A699894335557644%25401543879221725%2Fdownload%2FWhy%2BShould%2BEngineers%2Band%2BScientists%2BBe%2BWorried%2BAbout%2BColor_.pdf&usg=AOvVaw1qwjjGMd7h_z6TLUjzu7Nb [escaping-rgbland-pdf]: https://eeecon.uibk.ac.at/~zeileis/papers/Zeileis+Hornik+Murrell-2009.pdf [escaping-rgbland-doi]: https://doi.org/10.1016/j.csda.2008.11.033 <!--R Documentation--> [rdocs-extremes]: https://rdrr.io/r/base/Extremes.html [rdocs-range]: https://rdrr.io/r/base/range.html [rdocs-quantile]: https://rdrr.io/r/stats/quantile.html [rdocs-c]: https://rdrr.io/r/base/c.html [rdocs-list]: https://rdrr.io/r/base/list.html [rdocs-lm]: https://rdrr.io/r/stats/lm.html [rdocs-coef]: https://rdrr.io/r/stats/coef.html [rdocs-devices]: https://rdrr.io/r/grDevices/Devices.html [rdocs-ggsave]: https://rdrr.io/cran/ggplot2/man/ggsave.html [rdocs-dev]: https://rdrr.io/r/grDevices/dev.html <!--Wikipedia Links--> [wiki-snake-case]: https://en.wikipedia.org/wiki/Snake_case [wiki-hello-world]: https://en.wikipedia.org/wiki/%22Hello,_world!%22_program [wiki-janus]: https://en.wikipedia.org/wiki/Janus [wiki-nesting-dolls]: https://en.wikipedia.org/wiki/Matryoshka_doll [wiki-pure-fxns]: https://en.wikipedia.org/wiki/Pure_function [wiki-camel-case]: https://en.wikipedia.org/wiki/Camel_case [wiki-mojibake]: https://en.wikipedia.org/wiki/Mojibake [wiki-row-col-major-order]: https://en.wikipedia.org/wiki/Row-_and_column-major_order [wiki-boxplot]: https://en.wikipedia.org/wiki/Box_plot [wiki-brewer]: https://en.wikipedia.org/wiki/Cynthia_Brewer [wiki-vector-graphics]: https://en.wikipedia.org/wiki/Vector_graphics [wiki-raster-graphics]: https://en.wikipedia.org/wiki/Raster_graphics [wiki-dry]: https://en.wikipedia.org/wiki/Don%27t_repeat_yourself [wiki-web-scraping]: https://en.wikipedia.org/wiki/Web_scraping [wiki-xpath]: https://en.wikipedia.org/wiki/XPath [wiki-css-selector]: https://en.wikipedia.org/wiki/Cascading_Style_Sheets#Selector <!--Misc. Links--> [split-apply-combine]: https://www.jstatsoft.org/article/view/v040i01 [useR-2014-dropbox]: https://www.dropbox.com/sh/i8qnluwmuieicxc/AAAgt9tIKoIm7WZKIyK25lh6a [gh-pages]: https://pages.github.com [html-preview]: http://htmlpreview.github.io [tj-mahr-slides]: https://github.com/tjmahr/MadR_Pipelines [dataschool-dplyr]: https://www.dataschool.io/dplyr-tutorial-for-faster-data-manipulation-in-r/ [xckd-randall-munroe]: https://fivethirtyeight.com/features/xkcd-randall-munroe-qanda-what-if/ [athena-zeus-forehead]: https://tinyurl.com/athenaforehead [tidydata-lotr]: https://github.com/jennybc/lotr-tidy#readme [minimal-make]: https://kbroman.org/minimal_make/ [write-data-tweet]: https://twitter.com/vsbuffalo/statuses/358699162679787521 [belt-and-suspenders]: https://www.wisegeek.com/what-does-it-mean-to-wear-belt-and-suspenders.htm [research-workflow]: https://www.carlboettiger.info/2012/05/06/research-workflow.html [yak-shaving]: https://seths.blog/2005/03/dont_shave_that/ [yaml-with-csv]: https://blog.datacite.org/using-yaml-frontmatter-with-csv/ [reproducible-examples]: https://stackoverflow.com/questions/5963269/how-to-make-a-great-r-reproducible-example [blog-strings-as-factors]: https://notstatschat.tumblr.com/post/124987394001/stringsasfactors-sigh [bio-strings-as-factors]: https://simplystatistics.org/2015/07/24/stringsasfactors-an-unauthorized-biography [stackexchange-outage]: https://stackstatus.net/post/147710624694/outage-postmortem-july-20-2016 [email-regex]: https://emailregex.com [fix-atom-bug]: https://davidvgalbraith.com/how-i-fixed-atom/ [icu-regex]: http://userguide.icu-project.org/strings/regexp [regex101]: https://regex101.com [regexr]: https://regexr.com [utf8-debug]: http://www.i18nqa.com/debug/utf8-debug.html [unicode-no-excuses]: https://www.joelonsoftware.com/2003/10/08/the-absolute-minimum-every-software-developer-absolutely-positively-must-know-about-unicode-and-character-sets-no-excuses/ [programmers-encoding]: http://kunststube.net/encoding/ [encoding-probs-ruby]: https://www.justinweiss.com/articles/3-steps-to-fix-encoding-problems-in-ruby/ [theyre-to-theyre]: https://www.justinweiss.com/articles/how-to-get-from-theyre-to-theyre/ [lubridate-ex1]: https://www.r-exercises.com/2016/08/15/dates-and-times-simple-and-easy-with-lubridate-part-1/ [lubridate-ex2]: https://www.r-exercises.com/2016/08/29/dates-and-times-simple-and-easy-with-lubridate-exercises-part-2/ [lubridate-ex3]: https://www.r-exercises.com/2016/10/04/dates-and-times-simple-and-easy-with-lubridate-exercises-part-3/ [google-sql-join]: https://www.google.com/search?q=sql+join&tbm=isch [min-viable-product]: https://blog.fastmonkeys.com/?utm_content=bufferc2d6e&utm_medium=social&utm_source=twitter.com&utm_campaign=buffer [telescope-rule]: http://c2.com/cgi/wiki?TelescopeRule [unix-philosophy]: http://www.faqs.org/docs/artu/ch01s06.html [twitter-wrathematics]: https://twitter.com/wrathematics [robbins-effective-graphs]: https://www.amazon.com/Creating-Effective-Graphs-Naomi-Robbins/dp/0985911123 [r-graph-catalog-github]: https://github.com/jennybc/r-graph-catalog [google-pie-charts]: https://www.google.com/search?q=pie+charts+suck [why-pie-charts-suck]: https://www.richardhollins.com/blog/why-pie-charts-suck/ [worst-figure]: https://robjhyndman.com/hyndsight/worst-figure/ [naomi-robbins]: http://www.nbr-graphs.com [hadley-github-index]: https://hadley.github.io [scipy-2015-matplotlib-colors]: https://www.youtube.com/watch?v=xAoljeRJ3lU&feature=youtu.be [winston-chang-github]: https://github.com/wch [favorite-rgb-color]: https://manyworldstheory.com/2013/01/15/my-favorite-rgb-color/ [stowers-color-chart]: https://web.archive.org/web/20121022044903/http://research.stowers-institute.org/efg/R/Color/Chart/ [stowers-using-color-in-R]: https://www.uv.es/conesa/CursoR/material/UsingColorInR.pdf [zombie-project]: https://imgur.com/ewmBeQG [tweet-project-resurfacing]: https://twitter.com/JohnDCook/status/522377493417033728 [rgraphics-looks-tips]: https://blog.revolutionanalytics.com/2009/01/10-tips-for-making-your-r-graphics-look-their-best.html [rgraphics-svg-tips]: https://blog.revolutionanalytics.com/2011/07/r-svg-graphics.html [zev-ross-cheatsheet]: http://zevross.com/blog/2014/08/04/beautiful-plotting-in-r-a-ggplot2-cheatsheet-3/ [parker-writing-r-packages]: https://hilaryparker.com/2014/04/29/writing-an-r-package-from-scratch/ [broman-r-packages]: https://kbroman.org/pkg_primer/ [broman-tools4rr]: https://kbroman.org/Tools4RR/ [leeks-r-packages]: https://github.com/jtleek/rpackages [build-maintain-r-packages]: https://thepoliticalmethodologist.com/2014/08/14/building-and-maintaining-r-packages-with-devtools-and-roxygen2/ [murdoch-package-vignette-slides]: https://web.archive.org/web/20160824010213/http://www.stats.uwo.ca/faculty/murdoch/ism2013/5Vignettes.pdf [how-r-searches]: http://blog.obeautifulcode.com/R/How-R-Searches-And-Finds-Stuff/