I’ve spent months trying to understand climate change, the current state of mitigation strategies, and how I can make a difference. These are some resources that I’ve found helpful.
Positive climate-related newsfeed
Intro to the concept of stabilization wedges
Interactive demonstration of how different levels of climate change can affect us
Making our electricity grid “smart”
Climate change symposia at MIT
Tipping points we may encounter
A fantastic overview of the ways people in tech can help
A journey through the climate crisis, told through a diverse set of articles
Report on ways that technology can fight climate change
A blog post about largely non-professional approaches to fighting climate change
Y Combinator’s climate-specific page
A post on finding relevant companies, and an associated open database of organizations fighting climate change
Step-by-step guide for the job search process
Places to look for jobs:
Industries with most software jobs:
Should we use less violent language when talking about climate change?
While this article on facing extinction is immensely depressing and I don’t entirely agree with its assumptions, I like this quote about blame and anger related to climate change:
In a recent blog post, writer James Kunstler proposed a pithy theory of why humans chose each step of our path in history: “It just seemed a good idea at the time.” We plunged forward with each new way of doing things, each new invention, because it made life easier at the time. There was no intention to destroy ourselves.
…
As theoretical physicist Peter Russell mused in a podcast conversation with me in 2016, “What if we saw ourselves as a cosmic flame blooming in the universe and coming to its natural end?”
What if we forgave everybody everything?
A frequently cited example of climate nihilism that’s generally denounced by scientists
Some additional resources on why and how we must hold hope:
Footwear These changes take pressure off metatarsals and encourage them to spread.
Functional correction
Reducing inflammation/pain
Reducing pressure on the nerve
Taping strategies:
What’s in common between these strategies?
Why does taping work? (from comments on https://www.youtube.com/watch?v=Db9qgYPrku4)
If you wrapped the tape completely around with consistent stretch all the way thru then yes, it would compress the tissue. That’s why I incorporated anchor points on both ends with zero stretch, so the tape behaves as a cross fiber taping for increased circulation, which will create surface area by lifting the skin towards least resistance. away from the body.
There are a million articles that say roughly the same thing - metatarsal pads, orthotics, cortisone shots, surgery. Here are some more thorough and alternative views. They may or may not be reliable.
Some simple conservative treatments; more thorough than other sites
I like this article - thorough review of someone’s experience
This guy has a lot of ideas and opinions; he says you should always address all three of mechanical, inflammatory, and neurological issues.
A lot of alternative views here, with focus on massage/stretching.
]]>I’m starting off with a local script that I can run manually. The next step will be to automate it, but that’s a topic for another time.
This is my first foray into web scraping! I don’t know why it took me so long - it’s not particularly hard. Here’s my process.
Let’s get set up!
For starters, I’m using Python 3.6.9 and pip3 as a package installer. (I should probably get to the modern age and update to 3.8, but this works just fine.)
Since I’m trying something new, I did a bit of web sleuthing to find out how people typically do this sort of thing. Python seems to be a common choice, which makes sense - it’s a great language for quick scripts and munging data. The requests library is a clear choice for making HTTP requests, and Beautiful Soup is popular for pulling data from HTML.
Let’s go ahead and install those dependencies:
$ pip3 install requests
$ pip3 install beautifulsoup4
In fact, we can be good Python developers and pin those versions in a requirements.txt file. (I wouldn’t typically care for a little project like this, but I’d like to be able to deploy this to something like AWS Lambda. Being explicit about your dependencies is nice when you’re running stuff externally like that.) This file can go in the project root directory and looks like this:
beautifulsoup4==4.8.2
requests==2.23.0
We can install dependencies from that file by running pip3 install -r requirements.txt.
I’ll start out by playing around in IPython (pip3 install ipython). It’s my favorite way to play around with Python. I’ll also create a file to hold code that I want to keep. Let’s call it scrape.py. (People like using Jupyter Notebooks for this sort of interactive development, but I’ve never tried it and it doesn’t seem necessary in this case.)
Starting IPython is easy: just run ipython.
Great! We can start writing code now. First, I know that we’ll need to make an HTTP request to get the page we’re scraping. Let’s work that out.
The Python requests library is easy to use - docs are here. Let’s try getting the HTML for a page in IPython.
>> import requests
>> r = requests.get("http://www.google.com")
>> html = r.text
That’s some HTML! Now we need to choose the right URL to scrape. This project is automating something that I already do - making a search for a product I’m interested in. So let’s use the URL for that search. There may be a better way to scrape the site for products, but this is a decent place to start.
A big blob of HTML text isn’t particularly useful on it’s own. That’s where Beautiful Soup comes in. We’ll follow the quick start instructions in the docs and get “soup” for our page:
>> url = "https://www.rei.com/used/shop/search?q=magma%2030%20sleeping%20bag"
>> html = requests.get(url).text
>> from bs4 import BeautifulSoup
>> soup = BeautifulSoup(html, 'html.parser')
Now we have a soup object! We can run stuff like soup.a to get <a> tags in the HTML.
That’s a nice, self-contained piece of functionality there. Let’s make a function in the scrape.py file to hold that logic, so we can use it later.
(I had to google how to write a python function, yikes. It’s been a while.)
from bs4 import BeautifulSoup
import requests
def get_soup(url):
r = requests.get(url)
return BeautifulSoup(r.text, 'html.parser')
To use this in IPython, import the module: import scrape. When we make changes to the file, we need to reload the module. The simplest way to do that is restart IPython, but I usually prefer:
>> import importlib
>> importlib.reload(scrape)
Now things get a little tricky. What do we want to pull out of the page? How do we select the right elements, and make sure it won’t break when REI makes minor changes to the website?
Let’s use Chrome DevTools to examine the page. All of the products that come up in the search are in a <ul> list, and each <li> item has the class "TileItem”. Nothing else on the page has the class "TileItem", so that seems like a safe thing to filter by.
>> items = search_soup.find_all("li", class_="TileItem")
Now we have the correct items, but each of those is still a long blob of HTML. We just want the title of the item. Titles are located in a <span> with class "title". So, to get the title of the first item, we can run:
>> title = item.find_all("span", class_="title")[0].contents[0]
"Magma 30 Sleeping Bag"
Beautiful Soup is a powerful tool, and you can filter HTML in almost any way imaginable. The docs are pretty comprehensive. It’s a bit of a challenge to choose filters that are not too brittle, but otherwise it’s easy to use. I’m a fan.
Finally this is starting to come together. I’d like to run what we have as a script. To do this, we need a) our file to be executable, and b) an entrypoint.
To make the file executable, we need to put this line at the top: #!/usr/bin/env python3. Additionally, in the command line, we run chmod +x scrape.py to give us permission to execute the file.
There also needs to be an entrypoint that specifies the code to run. In scrape.py, let’s add this block at the bottom of the file:
if __name__ == '__main__':
# We'll replace this with the code we want to run.
print("This is a script!")
Now run the script!
$ python3 scrape.py
This is a script!
Now we can put it all together!
To do this, there are a couple of important decisions we need to make: program input and program output.
We could just be lazy and hardcode URLs in. But it’s nice to be able to edit code and the input URLs separately, especially if this is deployed to a service like AWS Lambda. You don’t necessarily want to redeploy your Lambda every time you want to look for a new product. So, for now, we’ll just put them in a file in the project root directory.
Open the file (use a context manager so resources are released appropriately).
with open('rei_used_gear_searches.txt', 'r') as file:
urls = file.read().splitlines()
Apparently we need to use read().splitlines() instead of readlines(), otherwise we’ll end up with \n at the end of every value. Good to know.
Hmmm…we could make this a bit better. Search URLs are pretty obtuse and hard to read. It’s not very user-friendly to spit out a bunch of URLs in the results. So let’s label them. A CSV of $LABEL, $URL should work.
urls = []
with open(filename, "r") as csvfile:
reader = csv.reader(csvfile)
for row in reader:
urls.append({"label": row[0], "url": row[1]})
The output:
>> urls
[{'label': 'magma30regular', 'url': 'https://www.rei.com/used/shop/search?q=magma%2030%20sleeping%20bag&size=REGULAR%20-%20RIGHT%20ZIP'}, {'label': 'altras', 'url': 'https://www.rei.com/used/shop/womens-footwear?brand=Altra&size=6'}]
Our source CSV file:
magma30regular,https://www.rei.com/used/shop/search?q=magma%2030%20sleeping%20bag&size=REGULAR%20-%20RIGHT%20ZIP
altras,https://www.rei.com/used/shop/womens-footwear?brand=Altra&size=6
We could just print the results of each search URL. That would be easy. But, from experience, I know that sometimes I want to know which products have been added instead of which products exist.
For example, perhaps I’m looking for a particular brand of shoe. I really like Altra shoes, for their zero drop and natural foot shape. There are a number of models I’d be interested in, but not all. I might do a search for them today, and not want any of the models currently available on the used gear site. In that case, I’ll want my tool to tell me if a new model has been added, so I can check it out.
With this goal, we need to come up with a way to tell if a product has been added since we last ran this tool. To do that, we need to save results somewhere, so we can compare results from this run with the last run.
Writing our results to a file is a simple way to do this. There are a couple things to consider:
Here’s the code for writing the results of one search.
def save_products_list(results, label, target_dir):
timestamp = str(int(round(time.time())))
filename = target_dir + label + '_' + timestamp + ".txt"
with open(filename, "w") as file:
[file.write(r + "\n") for r in results]
Now that we have a list saved, let’s run a comparison. This is a bit messy, but it’ll do the job. It runs once per search URL.
def compare_to_previous_products(current_products_list, product_label, target_dir):
# Get and sort all result files (so they're in order of timestamp)
sorted_results_files = sorted(os.listdir(target_dir))
# Pick out the result files corresponding to this particular search label
previous_files = [f for f in sorted_results_files if product_label in f]
with open(target_dir + previous_files[-1], 'r') as file:
previous_products_list = file.read().splitlines()
if sorted(previous_products_list) != sorted(current_products_list):
print("FOUND A DIFFERENCE FOR PRODUCT {} (from {})".format(
product_label, previous_files[-1]))
print("OLD: " + str(previous_products_list))
print("NEW: " + str(current_products_list))
Good enough!
It’s also worth trying to make this a bit robust to changes on the REI website.
"TileItem"? This could be a problem - then our list of items will just be empty and it’ll appear as if there are no products. Let’s use a handy feature of the website: when you do a search, it shows you how many results were found in a div that says “x results”. That div has the class "count”, and doesn’t exist if no products were found. So we’ll compare that to the number of "TileItem"s we found.
expected_count_div = search_soup.find("div", class_="count")
if expected_count_div is None:
print("No products found.")
else:
expected_count = int(expected_count_div.find("span").contents[0])
if len(items) != expected_count:
raise AssertionError(
"Error on page {}: found {} items, but page said there would be {}".format(
url, len(items, expected_count)))
What if the “x results” count div ceases to exist, or its class name has changed so we can’t find it? That would be okay if the TileItems are still around - the assertions we just wrote will fail. But what if the REI developers change both the count and TileItem class names? I’m going to be lazy and not solve this problem right now, but its worth noting.
<span> containing the item’s title? The code will blow up if that happens. Not elegantly, but we’ll know. This line will happily throw an error.
title = item.find_all("span", class_="title")[0].contents[0]
Besides that, there’s not too much to think about. We can clean the code up, handle some edge cases, etc. We’ll also liberally sprinkle in print statements so it’s clear what’s happening when the code runs.
So what do we end up with? Our first run is not particularly interesting.
$ python3 main.py
Opening search URL file rei_used_search_urls.csv
Found 2 search URLS
----------------------------
EXAMINING PRODUCT: magma30regular
Search URL: https://www.rei.com/used/shop/search?q=magma%2030%20sleeping%20bag&size=REGULAR%20-%20RIGHT%20ZIP&size=REGULAR%20-%20LEFT%20ZIP&size=REGULAR
No products found.
RESULT:
This is the first search for product magma30regular
----------------------------
EXAMINING PRODUCT: altras
Search URL: https://www.rei.com/used/shop/womens-footwear?brand=Altra&size=6
Found product Olympus 2.0 Trail-Running Shoes
Found product Torin 2.5 Road-Running Shoes
Found product Torin 2.5 Road-Running Shoes
Found product Paradigm 2.0 Road-Running Shoes
RESULT:
This is the first search for product altras
But when we run it the next day:
$ python3 main.py
Opening search URL file rei_used_search_urls.csv
Found 2 search URLS
----------------------------
EXAMINING PRODUCT: magma30regular
Search URL: https://www.rei.com/used/shop/search?q=magma%2030%20sleeping%20bag&size=REGULAR%20-%20RIGHT%20ZIP
Found product Magma 30 Sleeping Bag
RESULT:
FOUND A DIFFERENCE FOR PRODUCT magma30regular (from magma30regular_1582257372.txt)
OLD: []
NEW: ['Magma 30 Sleeping Bag']
----------------------------
EXAMINING PRODUCT: altras
Search URL: https://www.rei.com/used/shop/womens-footwear?brand=Altra&size=6
Found product Olympus 2.0 Trail-Running Shoes
Found product Torin 2.5 Road-Running Shoes
Found product Torin 2.5 Road-Running Shoes
Found product Paradigm 2.0 Road-Running Shoes
RESULT:
No difference found for product altras
Hooray! The REI Magma 30 sleeping bag that I’ve been waiting for is finally available used. This little tool is proving useful already.
Coming another day: running this automatically, probably via AWS Lambda.
See the proof-of-concept code here.
I made a lot of mistakes when first learning Java, and wanted to prevent others from falling in the same traps.
Some junior developers were starting to work on the Java codebase, and I ended up repeating a lot of the same advice in code reviews.
Everyone had differing opinions. That’s quite fine, but getting everyone to agree on a “correct” way of doing things leads to a much cleaner codebase. This required quite a bit of compromise, but it reduced friction afterwards.
I started out with scattered list of contentious points and common mistakes. I talked through the difficult points with other people, got reviews from interested parties, and put the final result in our developer docs. It was easy to direct developers to it when needed, and easy to append new points we wanted to specify.
It’s important for a software team to have agreed-upon guidelines like these, whether codified or not. Clearly there’s benefit to sharing knowledge of common mistakes. Code reviews get easier - less nitpicking required. It also keeps the peace. A team can spend endless time bikeshedding, returning to the same issues repeatedly, never leaving anyone happy. Instead, you can make the controversial decisions once and leave it at that.
Doing this allows the whole team to own the code. Codebases become difficult to read when everyone writes code according to their own preference. It’s difficult to predict where things are defined and reason about behavior. And it’s never comfortable to fiddle around with a class that someone else wrote in their own style.
This obviously should be a living document, in a team wiki or something similar. Making it easy to edit keeps it relevant to your ever-changing team and product.
As a side note: a great way to reduce the number of guidelines you need is to implement a style guide (for example, the Google style guide). It takes basically no effort to follow it if you use a formatter like Spotless. Similarly, you can use a static analysis tool like Error Prone to prevent people from committing certain classes of mistakes. I’ll also point out the Oracle Java Tutorial as a great resource for Java beginners.
With that, here are the guidelines that we used (edited a bit to be less domain-specific).
Write for readability over performance. The reason for this is two-fold:
The JVM JIT compiler is magic. In many cases, it can optimize your code better than you will, through branch predictions, loop unrolling, inlining methods, allocation elimination, lock elision, and much, much more.
Most performance “optimizations” implemented by the programmer have a very minimal effect, especially given the aforementioned compile optimizations. Writing for readability will likely save more time than the programmer’s optimization. This is both time spent by engineers trying to understand the code, and time spent chasing down obscure bugs.
Obviously there will be some complex situations when this guideline does not hold. Use your best judgement.
Use checked exceptions (ex. IOException, InterruptedException) for recoverable errors, and unchecked exceptions (ex. RuntimeException, AssertionError, NullPointerException) for unrecoverable errors. Both types of exceptions should have clear and descriptive messages.
Checked exceptions should be specific. In many cases it is useful to create a domain-specific checked exception.
Do not catch unchecked exceptions. Their role is not to be caught, it is to kill the program loudly. The exception to this is when programming external-facing APIs, where there may be certain expected behavior in the case of an exception.
Public methods and classes should always have a docstring that describes their role, expectations, and edge-case behavior. The format for this is often made clear in style guides. Documentation for simple getters and setters is not necessary. Document any private methods and classes if it will be useful for the reader.
Be careful with mutable class fields. Use immutable types when possible. Do not provide outside callers direct references to the field — return a copy, if practical. Encapsulate any mutation inside class methods, and consider what happens if multiple callers are simultaneously calling these methods.
Do not write fun and tricky logic. Boring logic is easy to test and easy to trust. If your logic is fun, separate it into boring and individually-tested pieces.
Handle nulls explicitly. If a field or argument may be null, use the javax @Nullable annotation to make the intention crystal clear.
Consider wrapping common calls to external libraries if they have undesirable, unpredictable, or poorly defined behavior. Common examples of this in Java can be found in OpenCV and the AWS SDK. This can save you from debugging the same issue repeatedly — put the messy library call in a method, test its behavior, catch exceptions and validate input as needed, and use that method in the future. And remember — if these methods aren’t easily discoverable and useable, or if the rest of the team finds them unnecessary, putting time into building a wrapper will be a waste.
[This seems like a silly thing to talk about at length, but we worked with a number of collections that needed almost the same interface/behavior as native Java collection types. There were weeks of heated debate over the correct approach to this, so we came to a decision and codified it. No one was quite happy, but it stopped being an issue.]
Do not extend native Java collection types (e.g. HashMap, ArrayList). It is very difficult to ensure that the new object will behave as expected when treated as the base type. Extending collection types is standard practice in some other languages, but Java programmers generally discourage it.
For simple data storage, an alternative is to store your desired collection as an attribute of another object. If you want to expose the collection methods, delegate them. If you want custom methods for your collection, you can write them on your outer class.
In more complex situations, consider implementing a collection interface (like Map instead of HashMap). This forces you to write all of your class’s methods, which makes behavior more predictable. (This strategy is an example of using the composition over inheritance principle.)
For asynchronous code, use CompletableFutures. They allow async computation to be performed without blocking a thread. One common use case is when making network calls, which involve dead time spent waiting for responses. There are admittedly drawbacks to CompletableFutures, but using them everywhere allows us to chain asynchronous callbacks, and more easily reason about async processes.
Choose composition over inheritance whenever reasonable. In Java, this means implementing interfaces instead of extending base classes. Implementing interfaces allows your classes to guarantee multiple behavioral contracts, instead of being bound to a specific base implementation that is not robust to refactoring.
Keep classes and methods small. Classes should represent simple entities. Methods should have one job. This is crucial for testing, refactoring, understanding, and maintaining.
Make class fields private, and provide getters when necessary. This is necessary to encapsulate the logic in a class, so callers are not dependent on the class implementation.
Avoid large blocks of code. Use whitespace to separate tiny, logical chunks of code. (Do not be afraid of whitespace. It is your friend.) Use inline documentation to describe unclear bits of code, but better yet, restructure your code and rename methods/variables until the code is clear.
Name your classes, fields, and methods descriptively. Auto-complete in an IDE removes the effort involved in using longer names, and clear names are what makes your code human-parseable.
When writing a piece of code, future readers should not be able to tell what member of the team wrote it. Having an institutionally-consistent codebase allows the whole team to easily read, understand, and alter unfamiliar code.
It’s impossible to have a complete list of guidelines — there are whole books written on this subject. However, if there is contention over an issue, or if something keeps tripping people up, it’s valuable to define what to do in that situation in the future.