Photo by Ferenc Almasi on Unsplash

How I completed my own coding project

Presentation

This article will describe how I realized a node project to get a regular account for my wallet of crypto currencies.

Realizing such small and personal project is important to become a better developer. It’s the occasion to learn new technologies, to try new methods and to test frameworks without consequences. Program creation is a knowledge that you can only learn by practicing, but in your professional career you don’t really have the occasion to progress. Personal projects are here to fill the gap.

In my own cases, I have some experience in javascript, and I have already done a few node projects. I wanted to progress with this tool, and that was my main motivation, beside the sheer utility of knowing how much my wallet worth.

The article itself is not so much about technologies. I will provide links to technological article I’ve read to fulfill my program, and I’m sure you will find good ones on your side too. No, this article is about how I made it, what was the difficulties, and how I solved them. It’s not here to explain the usage of a framework. Its purpose is to illustrate what happens when you try to really do something with your knowledge, how to find solution for problems that occur in real cases and never in perfect tutorial. In short, it’s about the real life of a developer.

All my code has been publish on github. You may find it here : https://github.com/FChenebit/CryptoWallet

Preliminary works and tests

So the first step was to have a rough idea of what I wanted to do. This idea should be summarized in one or two sentences. If longer, it’s probably a too large project to be confident you’ll finish it.

The summary for this project is: “I want a program that will run without me thinking of it, to send me once a day a recap of my wallet, the sum of its values, and the details for each currency”.

Usually, the second step would be to detail this summary. This would be the design phase (see next chapter). But as this is a project I was also using to learn; I knew I will have to master some points I don’t yet. And as I wanted to finish this project, I didn’t want to be blocked too long by something I would not be able to implement the way I wanted. My ignorance here was a blind spot: I may have tried to do something that is not possible or very difficult without knowing it. So, I must first have a better understanding of things I wanted to learn, and I would do some POC (proof of concept) to be sure I can complete my project.

• Cloud: If my program is to run alone, it must be on a cloud server. I’ve read some articles about Heroku, and I gave it a try. I see this is a PaaS platform and not a IaaS, so I’ll have less freedom than on my own server, but a lot of things are already deployed. I went through the documentation. My main problem was that Heroku suppose I’ll do a web site or a web application, that will be a server. Reading the get started documentation taught me a lot but didn’t help for my case. I needed a way to launch my node program at a given time. This is not a standard function, but Heroku has an add on able to do that: the Heroku Scheduler. I found it by searching schedule in the Heroku documentation, as I wanted to schedule a task. This add on documentation also pointed me to one off dyno that it will use, and how to run it in an Heroku console, to test it without waiting for schedule time. To test it worked, I’ve done a basic node program that will log a hello world message, and see it run in console and at scheduled time. Here, I was helped by my experience: I knew there should be a way to schedule some tasks, as it is a very common need. I may have tried to develop my own scheduler, but often, for common problem, there is a common solution, and your one will not beat it. It will not be as tested, you will not have as much experience, and many users have already fond all possible bugs. Trust me on this, don’t reinvent the wheel.
• Lint: I wanted my code to follow common best practices, to enhance readability and prevent some bugs. As I use Visual Studio, es lint seems to be the good choice. I’ve searched these keywords: “visual studio code eslint node.js », and the first answer looks fine: https://www.digitalocean.com/community/tutorials/linting-and-formatting-with-eslint-in-vs-code. After using it, it worked.
• Promises: when using JavaScript, I often failed into the callback hell: having to pass callback as parameter, and difficulties using it. Async module was a great help there, but I felt it was not enough. I’ve heard about promises, and I decided to use them in this project. I found this site during my everyday search: https://github.com/goldbergyoni/nodebestpractices and it talked about promises and async/await, but was not very clear. So, I searched with these keywords: “node 8 async await” (as async and await are node 8 features). I read the last one as this feature is quite recent. This is an internet pitfall: as there is not much recycling, you may find a lot of outdated information. This is even more important with node as it is a framework that is moving quite fast. To test my learning, I needed an async module.
• API: I needed an API to get crypto currency current rate. I will also use it to test async and await. The web site I used to buy my crypto doesn’t have such an API, so I searched: ”api to get COS/BTC”, as COS was one of the currency I have. I looked at some results and choose https://coinmarketcap.com/. To test different API, I used the postman tool. It’s very useful to send requests to understand how a REST API work. Coinmarket is easy to use and give me all the data I need. I made a test with async/await, following the example in the site above, and it worked just fine. I needed to choose a module to execute request. In my previous tutorial, I used request, but this module is deprecated. In the request project, a few alternatives are proposed. Got seemed the easier to use, and my POC confirmed that. So, there was no real need to find a better solution, as I had a good one that will be fine.

My POC for got

• Mail: the easiest part of the POC. A quick search shows me the module nodemailer. Like got, it does the job just fine.

With all these preliminary tests, I was very confident in my capacity to finish this project, as most of the technical difficulties have already been solved. I will probably have to set up a database, but this is a very common issue, so I’m sure I’ll find a solution. Now I “just” have to design exactly what has to be done, and to organize code to implement that design.

Photo by Daniel McCullough on Unsplash

Design

In a professional project, when you build an application for someone, you first describe with him what the application will do, in order to have an agreement. Then you detail how the application will be done (which language, which component…), then you do and test it. The first 2 steps are usually as detailed as possible. It may be a problem, as it is difficult to exactly know and describe what you do. If you want to build a house you may have architect plans, but they will be quite different from the real house. And describing a house is much easier that describing a program. Moreover, when describing what client want, you’ll quickly arrive to technical details he doesn’t want to hear about(client/server, do you need an API…). It shouldn’t happen, but in general technic has an impact on specification. That’s why agile methods are popular now: they only require the description of a short part of the program, to agree with the user before carrying on.

For a personal project, you can easily agree with yourself. So instead of a specification, you need a design. A design will detail your project, what you expect to realize. Then It will describe shortly the components of the program you’ll do. For such a short project, components will be files. For something larger, it will be packages. Then, write down the order you want to realize the components. The idea is to have a plan, a roadmap to the end of the project.

So with all that in mind, here is my design for crypto wallet:

Purpose

I want my program to send me every day a mail, that will give me the current amount of all my crypto currencies, and the detail currency by currency. This detail will also include variation on one week, one month, six month and a year. I also want that my program sends me an email if my wallet goes over a value threshold. It will test this alert condition every hour.

NB: at first, I thought sending normal email once every week, but it would have been difficult with the Heroku scheduler and having it once a day is not such a constraint. In a personal project, you can adapt what you want to technical constraint.

Functionality

With the purpose, I deduce the following list of functionalities inside the program, in no particular order:

• Get actual price of crypto currency with one or more API.
• Compute the total value of my wallet.
• Store my wallet details.
• Read the previous value of my wallet: one week before, 1 month before, 6 months before, one year before.
• Test if I go above a threshold.
• Read in a config the quantity of currency I own.
• Read the threshold in a config. (having hard constant in code is usually bad).
• Write alert email.
• Compute each line in email, with an evolution percentage.
• Write ordinary email.
• Send an email.
• Being able to send error.

Sending error is a vital component. You need your program to be able to tell you what he is doing, because sometimes, it can be the only way to debug.

Components

With the functionality, I can list all the components of the program:

• Logger: expose a log function, that will test environment. If environment is development (ie on my own computer), then write to console log. If production (ie on Heroku), then send an email.
• Mail: expose sendMail, that will send an email with info taken in the config. Its parameters are subject and email text.
• Scheduler: will order all the other call and be called by Heroku.
• Request: to get data from the API tested in POC.
• Database: will get history data from a database. Take care, I may not have some date (as I have to use program one year to have a one-year history for instance).
• Compose: to write normal and alert email.

The first 2 modules to develop are Mail and Logger. As said above, it’s vital to have the ability to log messages to understand what’s happening in your program, so a logger should always be your first module. It’s also why hello world is the usual first example in any language: to know our program has run, you need an output.

Next, there are 2 options: follow the process order (request, database then compose or down top), or do it in the reverse order (or top down). With top down, you may be wrong when guessing interface of function not already done, with down top, you may do things you’ll not use, and loose the global vision of the software. In that case, there is not so much to do, and the risk of doing useless things is reduced. Also, I don’t have so much freedom in data and so in parameters. So down top will be my choice.

So final order is: Mail -> Logger -> Request -> Database -> Compose -> Scheduler.

Photo by Stanislaw Zarychta on Unsplash

Setting up my environment

After design, I needed to set up the tools I’ll use to develop crypto warning.

• IDE: I’ll use Visual Studio. Its configuration is quite easy. The only point was using ES-Lint, but I explained it above.
• Source Control: a source control tool is quite mandatory when many people work in the same project. If you’re alone, it may seem less useful. For my part, I see 2 interest in source control: it saves your code on a server, and so you will never lose it. You can also easily work on another computer. The other point is history: if suddenly your code has a problem, you can check how it was modified to help you understand what did happen. I need 2 repositories: one for Heroku because it is the way to deliver code on their platform, and one for github as a support for this article. I don’t really know how to do, so I proceed with a try. Heroku document suggest to create an Heroku app in an existing folder, and Heroku will add this folder to a repository it manages. I tried that on a folder that already have a repository, and it worked just fine. SourceTree shows me the 2 distant repositories. I tried this before starting to code, in case I loose everything. Fun fact, I had a problem with SourceTree that seems related to authentication. The message was not clear. (it is very important for all messages to be clear and to give all available information. Reader will sort what he needs, don’t try to do it for him). It worked in console mode. After some try, I closed the window of the project in SourceTree, and then I see an error message hidden by that window. This message asked me to do some action on github, and after that it worked. Sometimes, modal windows are messy on Mac.

SourceTree, a very useful tool for source control

• Time: I decided to work on hour each evening, after my baby is asleep. It’s very important to be regular to not loose motivation here.

With everything ready, it’s time to code.

Realization

Mailer

With the POC I’ve done, the only missing point was promise. The nodemailer documentation explain that it natively support promises. I just need to omit callback parameter, and sendMail function will return a promise.

So my wrapper will only add parameters that will never change: SMTP host and port, user and password, from and to address. These parameters will be put in configuration. On Heroku, this configuration will be set in config vars in the setting. To test locally, I put the same config var in a shell script that will then run my node program. Of course, this shell script was not committed. Here is what it looks like:

I tested it locally and on Heroku with Ok test (to send email), and error test (with missing config vars). I will not write that in next chapters, but I’ve done such test for all modules.

Corresponding commits are 0ede84f and 3bdfa6d.

Logger

This module is also a very simple one. I test a config var to know if I use local or Heroku (DEV or PRODUCTION value).

Corresponding commits are 9c1a9fb and 1af355b.

Request

This was the first module to be a bit more complicated, but most of the work was done in the POC. I used got in a promise. I just wrapped around it the request I’ll do. I used a constant for the URL instead of a config var, as I will never change it. I was afraid to have problem with error management, but the magic of async handle it perfectly. I used https://jsonlint.com/ to analyze the result of the request. I believed I had a problem on Heroku, as I had result in console but not email. But problem also occurred locally, and I didn’t understand that (It’s very easy to misunderstand log message, and search in the wrong place). The problem is an error LPN105_510 when sending an email. Internet doesn’t know this error. I supposed this an anti-spam filter. I left it for later.

Database

I needed data, to have some history about the values. my first thought was to save these data in a file, as volume will be very low. (remember, my moto is to keep things as simple as possible). Unfortunately, Dyno (the server on Heroku) don’t keep file between each run, so I need a database. Mongo could have been a good choice, but there is no free add on for Heroku to use it. So I decided to use a PostGreSQL database. A POC could have been a good idea, but at that time I didn’t thing I would need a database.

Heroku documentation on PostGreSQL is quite clear. I need an add on for Heroku, and a local database. I will have only one table, called currency_rate, with the currency code, the date of the rate and the value of the rate. I installed pgAdmin to have an easy access to my database, but this tool wasn’t very efficient. I decided instead to work in console, as I don’t have too much task : I needed to create a table, and some test data. I had an error in my table creation, as I used CamelCase for the table name (CurrencyRate). With underscore it worked (currency_rate).

I first tried to have a basic request working: select NOW();. But it didn’t work, my program needed to be killed. (The problem is in commit 1943387). I first thought it was a problem with promises, but not. So I tried to start and end my connection to the database. It doesn’t work either, but with a new error message I see where my problem was. In needed to start and end a connection to my database, but if I export client.connect() instead of return client.connect(), I don’t return a promise, I execute it immediately. This correction is done in commit 92470a0.

I still had a problem on Heroku, but I see in the documentation that my configuration is different: rejectUnauthorized is set to true on development, and I need to set it to false on Heroku. To do this, I use the ternary operator. It’s an unusual operator, but it can greatly help you in such case.

I prefer not to use exotic features, as they make the code less easy to understand, but this one is very useful, and I make an exception for it.

As now my request work, I will use my test cases. I set them in the first commit for the database (1943387). I tried to have many different cases, and I wanted them to be easy to distinguish from each other, to not make mistake when I will see them in my tests. As I am not sure the rate I’m searching is available at the date I’m looking at, I will take the next available date. We’ll see later this was a mistake.

Also, take care to bind your parameters to your query. You can just build a string and add them inside this string, but binding will do some additional checks that will prevent conversions errors. These errors will especially happen with date type data.

Trou Normand

Photo by Sidral Mundet on Unsplash

Before going to composer module, I’ve taken a long break. In fact, I’ve been doing an online training that was planned for a long date. This break stops my project for quite long. As I knew I would have stop, it would have been a better strategy to finish composer before database and have a daily mail without history. I would have something useful I could have completed later. Here, I’ve taken the risk to stop my project and to never continue it. It didn’t happen but taking the risk was a mistake.

Also, I wanted to solve my mail problem. After some research on internet, I found that someone with the same error suppose an anti-spam problem, without proof. As I have the problem with my first sending, this is not because I send too much mail. And I’ve been able to send a few, so problem is elsewhere. I modify the text sent, and It worked. After some test and error, I’ve found I got the error when my mail text is only a JSON. I suppose the reason is that a JSON mail is recognized as a spam. With plain text, it’s working. So, in case of error, I will just log it in Heroku, and send a mail with a text message telling me I’ve got an error and that I must check my logs.

Composer

I was then ready to carry on the main track. My problem here is that Composer was not clearly designed. I know what it had to do, but I didn’t think enough about how. But as my modules are independent, I can do it now.

I wanted in my mail a line by currency, with actual and all historical value. I can easily get actual value by currency from request, but from database, I’ve got all date from my history and then each currency. I needed to “turn” this result. So my algorithm will look like:

1. Get actual data from request
2. Get historic data from database for one week history
3. Get historic data from database for one month history

· …

4. Compute Total value from request result

5. Write total result in a string

6. Turn database result

7. For each currency, write actual value, and historical, then insert actual value for future mails.

8. Send the string written in an email

So I changed my design: my schedule module will implement the above algorithm. Its new purpose is to schedule the execution of request. A module compute will replace compose, and will compute total actual value, and turn the results from the database. Scheduler module will be my starting point of my program. I kept index.js for testing purpose.

I’ll first do all that is related to actual value, and then work on history. The first task is quite easy : I put the quantity of crypto currency I own in environment variable. You may read it in commit beea66a. I tested that I got an error message when one variable is missing. It worked locally but not on Heroku. I’ve done some test, with log message. As expected, problem was in these environment variables. It was expected because that’s the difference between the 2 environments. In fact, I put them on the wrong Heroku server.

The second part was a bit more difficult. My first problem was I may not have the rate at the date I wanted, because program may not run every day. So, I defined an interval for each historic research: 3 days for one week, 7 for one month… I first thought that I will use this interval in compute or schedule, but it was a bad idea. The best way is to change the database module: instead of one date, I use 2 in parameters, and take the first date between the 2 in parameters. With that I’m quite sure I have the history data if they exist, even if not in exact day.

Turning database result was quite easy, as soon as I understand what to do. I have from database something like that: database -1W = [{currency_code,rate_value,rate_date}] ; and I wanted something like that : [ {BTC : {6M:Rate,1W:Rate}… I checked with the debugger my understanding of returned structure is correct (log message could also have done the trick here).

I’ve just got a problem in scheduler, as the foreach(currency) loop call an arrow function that is not async, so It couldn’t call the database promise. I solved it by creating the promises in the foreach loop, and then call them after the loop.

All these modifications are in the commit: af8a0ce.

Alert mail was quite easy too. I use request and compute to get actual value, and if this value is above a threshold, I send a basic email message with this information. My last task was to remove index.js that was not used anymore.

Conclusion

At the end of this project, I’m happy to have been able to finish it. What gets to the finishing line was 2 points:

• Each task to do was a small step, with a quick success.
• I had a general roadmap to follow, so I knew where to go to finish.

In fact, your working method has only 2 questions to answer: what will I do next, and how will I finish, what is the next step, and what is the plan to the end.

As long as you’ve these 2 answers, nothing can stop you.