Installing programs on Linux from command line.

To learn how how to install new programs on your Linux machine we will start by examining a preinstalled program called top.

The program top provides a real time dynamic view of resource usage and currently running processes. It can be invoked by the command top on the command line.

Typing top into the command line of your terminal starts the top application. The upper part is a summary of resource usage and the bottom part under the black line in the screenshot is the running processes.

You even see here that the user, ec2-user , is running top because I am!  The top part is showing me a summary of how many processes are running, how much CPU I am using and how much memory is available and being used, and the bottom part is breaking the usage out by each process. Pretty awesome.

If you played along and you now have this list of processes running on your Linux machine, you will want to get back to the command line prompt by simply typing “q” for quit.

This is not an exhaustive tutorial on top. The top program is used here so I can show you a few things that will help you when you install your first program on your Linux machine or EC2 instance.

OK I will show you one top specific command.

Typing

$ top -a 

will sort the top process list by memory so you can see which processes are taking up the most memory. This is very important in server management.

In the command  top -a the command is top and -a is an option you can set on the command. You can find options by reading the documentation on top.

Let’s explore top a little bit. This program came prepackaged but where is top “installed”? Another way to ask that is, “where is top’s executable binary?”

Linux has a few other commands that can help us answer that questions.

We have three pretty good options:

  • whereis
    • whereis locates the source, binary and manuals sections for specified files
  • which
    • which locates the executable file associated with a given command
  • find
    • find searches for files in a directory hierarchy

You might be thinking, I thought this tutorial was about installing new programs on a Linux machine. It is! Bear with me. I am hopeful the stuff I am explaining will save you a lot of headache in the future.

The goal of this tutorial is to install your first program on your Linux machine. We can get into specifics later let’s actually do what we came to do.

To install our first program, htop, we are going to use the package manager, yum. Htop is a interactive process viewer for Linux like the preinstalled top, but is a bit more visual and provides more information than top. Htop is a great system monitor.

$ sudo yum install htop

Now simply type htop into the command line.

$ htop

$ which htop
/usr/bin/htop

You can see that htop is the in the same directory, /usr/bin/,  as top. 

When we check the permissions of htop we see that the two executables have the same permissions, group and users.

ls -la htop top

-rwxr-xr-x 1 root root 121312 Jul 24  2013 htop

-rwxr-xr-x 1 root root  58400 Mar 17  2015 top

 

The permissions read -rwxr-xr-x

file –

Read Write Execute

Read   –        Execute

Read  – Execute

So any user on the system can execute this file.

$ echo $PATH

/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/opt/aws/bin:/home/ec2-user/.local/bin:/home/ec2-user/bin

 

usr/bin is in your path so you can execute htop simply by running the command htop anywhere by any user in the system.

Analyzing the permission, groups, owner and path will be important in running programs after you install them.

 

 

 

Why you need an Amazon Elastic IP Address.

If you are using Amazon Web Services and EC2 instances, go ahead and setup an Elastic IP address.

I am just going to save a you a little time here. We have chosen to use Amazon Web Services (AWS) because it is easy to create and terminate free Linux based servers thought their EC2 instances. We can try things out, and if I need to start over we can just terminate the EC2 instance and start again. Of we can quickly back it up and revert back to a previous state if we become frustrated.

However, my hope is that you get to a point where you are building an application you are interested in continuing. Inevitably you you’ll need to stop and restart you EC2 instance.

Whenever you start your EC2 instance it is assigned a public IP address and a public DNS. The public DNS resolves to the public IP address. You will be using these to connect to the server, and maybe to request your web application.

However, if you stop and restart this server, both the public IP address and the public DNS will change. You will then have to go update these wherever you have used them. It could be in your SSH command or your SFTP settings or in your local hosts file you are using to resolve virtual domains on your Apache server.

Regardless, it is annoying but there is often a price to pay for convenience, and it is super convenient to be able to spin up free computers whenever you need them.

We get around this by using Amazon’s Elastic IP addresses.

Fix: virtualhost overlap on port 80 the first has precedence

I had to virtual host conf files but only the first one was resolving

When I would restart Apache I would see this
 virtualhost overlap on port 80 the first has precedence


    ServerName localhost
    DocumentRoot /var/www/html/000




 and 


    ServerAdmin carla@localhost
    DocumentRoot /var/www/html/mysite.com
    ServerName mysite.com
    ServerAlias mysite.com
 This fixed it # NameVirtualhost is needed in httpd prior to v2.3.11 NameVirtualHost 192.168.0.1:80 
 NameVirtualhost *:80
    ServerName localhost
    DocumentRoot /var/www/html/000



I believe this NamedVirtualhost can be enabled in main config

Add User to EC instance with /home directory


[ec2-user@ip-172-30-1-107 conf.d]$ adduser paul
-bash: /usr/sbin/adduser: Permission denied
[ec2-user@ip-172-30-1-107 conf.d]$ sudo adduser paulpend
[ec2-user@ip-172-30-1-107 conf.d]$ cd /home
[ec2-user@ip-172-30-1-107 home]$ ls
ec2-user paulpend
[ec2-user@ip-172-30-1-107 home]$ ls -la
total 16
drwxr-xr-x 4 root root 4096 Feb 17 23:21 .
dr-xr-xr-x 25 root root 4096 Feb 16 19:22 ..
drwx------ 4 ec2-user ec2-user 4096 Feb 17 07:35 ec2-user
drwx------ 2 paulpend paulpend 4096 Feb 17 23:21 paulpend
[ec2-user@ip-172-30-1-107 home]$

 

Now this user needs access in someway

[ec2-user@ip-172-30-1-107 home]$ cd ec2-user
[ec2-user@ip-172-30-1-107 ~]$ ls
[ec2-user@ip-172-30-1-107 ~]$ ls -la
total 36
drwx—— 4 ec2-user ec2-user 4096 Feb 17 07:35 .
drwxr-xr-x 4 root root 4096 Feb 17 23:21 ..
-rw——- 1 ec2-user ec2-user 1262 Feb 17 07:35 .bash_history
-rw-r–r– 1 ec2-user ec2-user 18 Aug 30 19:00 .bash_logout
-rw-r–r– 1 ec2-user ec2-user 193 Aug 30 19:00 .bash_profile
-rw-r–r– 1 ec2-user ec2-user 124 Aug 30 19:00 .bashrc
drwx—— 2 ec2-user ec2-user 4096 Feb 16 19:22 .ssh
drwxr-xr-x 2 ec2-user ec2-user 4096 Feb 17 02:46 .vim
-rw——- 1 ec2-user ec2-user 996 Feb 17 07:35 .viminfo
[ec2-user@ip-172-30-1-107 ~]$ cd .ssh
[ec2-user@ip-172-30-1-107 .ssh]$ ls
authorized_keys

ec2-user etin with keys

 

http://www.linfo.org/su.html

 

If I try to go into this new users home folder I get permissioned denied.

Why?

Because I am currently ec2-user

 


[ec2-user@ip-172-30-1-107 home]$ cd paulpend
-bash: cd: paulpend: Permission denied
[ec2-user@ip-172-30-1-107 home]$ whoami
ec2-user

I can use the su command to change my login session to paulpend

 

However paulpend has no password


[ec2-user@ip-172-30-1-107 home]$ su paulpend
Password:
su: Authentication failure

So what the fuck should I do?

Well, since ec2-user is a sudoer then I can sudo and act on behalf of root who can do whatever they want.

[ec2-user@ip-172-30-1-107 home]$ sudo su paulpend
[paulpend@ip-172-30-1-107 home]$

Or alternatively I could get paulpend a password if I wanted

[paulpend@ip-172-30-1-107 home]$ pwd
/home
[paulpend@ip-172-30-1-107 home]$ ls -la
total 16
drwxr-xr-x 4 root root 4096 Feb 17 23:21 .
dr-xr-xr-x 25 root root 4096 Feb 16 19:22 ..
drwx------ 4 ec2-user ec2-user 4096 Feb 17 07:35 ec2-user
drwx------ 2 paulpend paulpend 4096 Feb 17 23:21 paulpend
[paulpend@ip-172-30-1-107 home]$ cd paulpend
[paulpend@ip-172-30-1-107 ~]$ ls -la
total 20
drwx------ 2 paulpend paulpend 4096 Feb 17 23:21 .
drwxr-xr-x 4 root root 4096 Feb 17 23:21 ..
-rw-r--r-- 1 paulpend paulpend 18 Aug 30 19:00 .bash_logout
-rw-r--r-- 1 paulpend paulpend 193 Aug 30 19:00 .bash_profile
-rw-r--r-- 1 paulpend paulpend 124 Aug 30 19:00 .bashrc
[paulpend@ip-172-30-1-107 ~]$ touch index.html
[paulpend@ip-172-30-1-107 ~]$ ls -la
total 20
drwx------ 2 paulpend paulpend 4096 Feb 17 23:37 .
drwxr-xr-x 4 root root 4096 Feb 17 23:21 ..
-rw-r--r-- 1 paulpend paulpend 18 Aug 30 19:00 .bash_logout
-rw-r--r-- 1 paulpend paulpend 193 Aug 30 19:00 .bash_profile
-rw-r--r-- 1 paulpend paulpend 124 Aug 30 19:00 .bashrc
-rw-rw-r-- 1 paulpend paulpend 0 Feb 17 23:37 index.html
[paulpend@ip-172-30-1-107 ~]$ nano index.html
[paulpend@ip-172-30-1-107 ~]$ ls -la
total 24
drwx------ 2 paulpend paulpend 4096 Feb 17 23:37 .
drwxr-xr-x 4 root root 4096 Feb 17 23:21 ..
-rw-r--r-- 1 paulpend paulpend 18 Aug 30 19:00 .bash_logout
-rw-r--r-- 1 paulpend paulpend 193 Aug 30 19:00 .bash_profile
-rw-r--r-- 1 paulpend paulpend 124 Aug 30 19:00 .bashrc
-rw-rw-r-- 1 paulpend paulpend 28 Feb 17 23:37 index.html
[paulpend@ip-172-30-1-107 ~]$

 

Now I’ve made an index.html file and I need Apache to server it.

Let’s try to make a virtual host.

Important Apache Files and Directories

We have already learned that Apache is configured by default to deliver web files from

/var/www/html

Here are the locations of other important files and directories

Apache Important Files and Directories

  • The default server root directory (top level directory containing configuration files): /etc/httpd
  • The main Apache configuration file: /etc/httpd/conf/httpd.conf
  • Additional configurations can be added in: /etc/httpd/conf.d/
  • Apache virtual host configuration file: /etc/httpd/conf.d/vhost.conf
  • Configurations for modules: /etc/httpd/conf.modules.d/
  • Apache default server document root directory (stores web files): /var/www/html

 

From the command line we can see


[ec2-user@ip-172-30-1-107 httpd]$ ls
conf conf.d logs modules run

Looking a bit closer by applying the -la flag to ls


[ec2-user@ip-172-30-1-107 ~]$ cd /etc/httpd
[ec2-user@ip-172-30-1-107 httpd]$ ls -la
total 16
drwxr-xr-x 4 root root 4096 Feb 16 21:18 .
drwxr-xr-x 79 root root 4096 Feb 16 21:18 ..
drwxr-xr-x 2 root root 4096 Feb 16 21:18 conf
drwxr-xr-x 2 root root 4096 Feb 16 21:18 conf.d
lrwxrwxrwx 1 root root 14 Feb 16 21:18 logs -> /var/log/httpd
lrwxrwxrwx 1 root root 24 Feb 16 21:18 modules -> /usr/lib64/httpd/modules
lrwxrwxrwx 1 root root 14 Feb 16 21:18 run -> /var/run/httpd

Setting Up Apache on Amazon EC2

 

Install Apache

 

We’ve installed Apache, a web server but we still can’t see a web page.

What gives?

This is because you can’t accept inbound traffic. You are allowing inbound traffic on port 22 for SSH but not on port 80 needed for HTTP.

So we need to allow Inbound traffic from Port 80

And now when we visit our public domain we see the default web page of the Apache HTTP server.

This page tells us the default location to add HTML content

“You may now add content to the directory /var/www/html/. Note that until you do so, people visiting your website will see this page, and not your content. To prevent this page from ever being used, follow the instructions in the file /etc/httpd/conf.d/welcome.conf.”
So let’s go there.
Currently we are in the users home directory. We can use pwd to see where we are.
You are in the home directory of the ec user
[ec2-user@ip-172-30-1-107 ~]$ pwd
/home/ec2-user
Let’s take a peek in this directory using the ls -la command. This will list all files and their group and user permissions. We will use ls -la again in a minute.


[ec2-user@ip-172-30-1-107 ~]$ ls -la
total 24
drwx------ 3 ec2-user ec2-user 4096 Feb 16 19:22.
drwxr-xr-x 3 root     root     4096 Feb 16 19:22..
-rw-r--r-- 1 ec2-user ec2-user   18 Aug 30 19:00 .bash_logout
-rw-r--r-- 1 ec2-user ec2-user  193 Aug 30 19:00 .bash_profile
-rw-r--r-- 1 ec2-user ec2-user  124 Aug 30 19:00 .bashrc
drwx------ 2 ec2-user ec2-user 4096 Feb 16 19:22.ssh

We need to go to the directory /var/www/html/
Let’s go check it out

[ec2-user@ip-172-30-1-107 html]$ cd /var/www/html
[ec2-user@ip-172-30-1-107 html]$ ls
[ec2-user@ip-172-30-1-107 html]$ ls -la
total 8
drwxr-xr-x 2 root root 4096 Feb 16 21:41 .
drwxr-xr-x 6 root root 4096 Feb 16 21:18 ..
We see there are no files here. Let’s create an HTML file, index.html using nano

[ec2-user@ip-172-30-1-107 html]$ nano index.html

Oh nooooooo permission denied (see message at bottom of next image).

What the hell. Well its because the permission for this directory is set to root, and we are logged in as ec-user.

Have you noticed in our command line

 

[ec2-user@ip-172-30-1-107 html]$

 

If we want to create a file in this directory as is, we need to write the file with root permissions.

You NEVER want to log in as root, and it is really hard in an Amazon EC2 server without some making some serous changes. Fortunately we can use the sudo command to do this.

If you add the prefix sudo to most linux commands, that command will run with elevated privileges required to perform certain, usually administrative tasks.

Basically if you need to do shit as root then instead of logging in as root use the sudo command.

The ec2-user has sudo privileges on AWS EC2 servers.

So a quick aside as I show you.

Sometimes you can find the list of users who can sudo here

less /etc/sudoers

However, on this Amazon EC2 instance the file is located here

/etc/sudoers.d/cloud-init

and we don’t have permission to see it as ec2-user

This is a bit meta but if you need to elevate your privileges with sudo to read the contents of the sudoers file.

[ec2-user@ip-172-30-1-107 etc]$ sudo nano /etc/sudoers.d/cloud-init

When the files opens you will see ec2-user listed.

A side note, you may have tried to use the change directory command, cd to get into the sudoers.d directory; however, you would get a permission denied as this directory requires root. You can’t use sudo cd because cd is part of the shell. sudo foo means run the program foo as root. cd is not a program  but an in-built command and sudo only applies to programs.

If for some reason you really wanted to cd into this directory you’d have to become root (this is NOT recommended)
[ec2-user@ip-172-30-1-107 etc]$ sudo -s
[root@ip-172-30-1-107 etc]# cd sudoers.d
[root@ip-172-30-1-107 sudoers.d]# ls
cloud-init

To get out of root and back to ec2-user simply use exit

[root@ip-172-30-1-107 sudoers.d]# exit

[ec2-user@ip-172-30-1-107 etc]$

See how the username before the @ changes from root to ec2-user

Phew ok back to making out first HTML page.

All of this to say that ec2-user has sudo privelges and as such can create an index.html with root permission to match the permssions of the direcotry that have the Apache HTML files
[ec2-user@ip-172-30-1-107 html]$ cd /var/www/html
[ec2-user@ip-172-30-1-107 html]$ ls
[ec2-user@ip-172-30-1-107 html]$ ls -la
total 8
drwxr-xr-x 2 root root 4096 Feb 16 21:41.
drwxr-xr-x 6 root root 4096 Feb 16 21:18..
[ec2-user@ip-172-30-1-107 html]$
OK so we are in the directory Apache told us the files are in, we understand the containing directories have root permissions. So we will sudo and make this file
[ec2-user@ip-172-30-1-107 html]$ sudo nano index.html

Control X will exit and ask you to save.

Navigate back to the public url for this EC2 instance. Boom!

 

 

410 days sober: Change over time.

Late 2013. Jonathan and I had been trying to grab lunch for awhile. We hadn’t seen each other in months. He is one of the smartest guys I know. The kind of person you listen to. Growing up he was both a brainiac and an athlete, but like me, gained a lot of weight as an adult. As he walked up the sidewalk I barely recognized him. He looked 20 years younger and 50 lbs lighter. He read a book called “Eat to Live” and eating a plant-based diet.

Dec 31st 1013 A slammed back my last whiskey for the night and wrote a note to tell the bartenders at Kaleidoscope I would miss them. I think I actually wrote “fuck y’all,” but they knew what I meant. I had decided to not drink for the next 90 days. You know, to “prove” I wasn’t an alcoholic and to try to lose weight.January 2014 I searched for the the most beginner yoga class I could find. I committed myself doing a Basics of Ashtanga yoga class at Balance Yoga. I was introduced to yoga by a girl I dated in graduate school. We did not have such things in the small rural Georgia town I grew up in. Search engine optimization (SEO) and love led me to Ashtanga yoga.

Over the course of the next few months:

I did not drink, practiced yoga 2-3 times a week, ate a plant-based diet and walked 2 miles twice a week. Atu and I played racquetball once a week. I lost 26 lbs in 90 days.

Ninety days later, on March 31st, my birthday, I started drinking again. I fortunately did not give up yoga. I credit the loving community at Balance Yoga.

The next few years I persisted on a mostly vegetarian(ish) diet. I ate eggs at Waffle House most mornings. Most evening I had whiskey, brussel sprouts and pimento cheese. I was injured a lot. Drunk a lot. And unhappy a lot. I got down to 182 lbs but was not healthy. I plateaued in life.

Jan 1st 2017. I stopped drinking. I follow the Ashtanga method of yoga and practice Mysore style 5 days a week. I seem to have an aversion to led primary. I try to eat healthy, but eat more sugar than I’d like. I read and write. I am happy, healthier and in love with my life. My goals and practice are now spiritual and philosophical; fitness is just a side effect.

Day 398: Do the Art Thing.

398 days sober….

I started the day with a hangover. Unfortunately this was not uncommon back then. I had to “teach”. Today would be a movie day.

I enjoyed teasing my students with music, movies and people outside of their pop culture knowledge. We had “guess the song” pop quizzes that featured 90 rap, 80s music and golden country. With my head pounding and looking for something somewhat science based, I happened upon “How William Shatner Changed the World”. They would meet Captain Kirk.

A thesis in this movie was that much of the technology we enjoy today was first “invented” by writers and set designers who constructed technology out of words and carboard boxes. This on-screen technology etched in some kid’s mind, and they dreamed of building it one day.

Years later I was invited to ideate with some really smart people for friend’s new business concept. Their professions ranged from art, design, policy to technology. Chris Appleton of WonderRoot and I openly discussed the differences and similarities in how artists and software engineers think and work. I realized how little I knew about an artist’s work, but something seemed familiar and akin to why I enjoyed programming.

Artists can “build things” way before the technology exists. I began to question just how fictive our world is.

I set out to learn more about the local art community. I didn’t know where to start. Nevertheless I set out. My plan was to engage with Chris and WonderRoot, but along the way I found BURNAWAY. I joined BURNAWAY’s board knowing very little about art. I wondered how my country ass would fit in.

I was at opening at Kibbee Gallery There was this damn pink cat with big green eyes. I was mesmerized by this piece but was afraid to buy it. I didn’t know if it was the “right” type of art to buy. I worried if other people would like it. The same themes that have haunted me my entire life.

Preston approaches me. “Butler if you like it, buy it. It is obvious you do. There is no requirement other than if YOU love it or not.” I bought my first piece of art that night.

Tonight I am helping host BURNAWAY’s Art Crush. I hope to see you there. Come as you are.

NO whiskey was consumed in the recreation of this event.

Drupal Apache Solr Module Search Results Missing Title

After finally getting my Apache Solr EC2 instance connected I was able to index and search for results on my Drupal 7 website except low and behold, when I did a search the search results appeared without titles.

 

The first thing I did was look in the search result template file to see if the title was being called properly.

It was.

So I had to figure out why the title variable might not be populating.

I tuned on visible error logging for this Drupal site and saw a number of errors having to do with arrays.

Error messages give you all sorts of hints. I then dove into the module code to try to decipher what was going on. Why not start right where the error message says around line 1372 of the file, apachesolr_search.module

 

Here is the offending code:

function apachesolr_search_process_response($response, DrupalSolrQueryInterface $query) {

$fields = (array) $doc;
// Define our url options. They depend on the document language.
$url_options = array(‘absolute’ => TRUE);
if (isset($doc->ss_language) && isset($languages[$doc->ss_language])) {
$url_options[‘language’] = $languages[$doc->ss_language];
}
$result = array(
// link is a required field, so handle it centrally.
‘link’ => url($doc->path, $url_options),
// template_preprocess_search_result() runs check_plain() on the title
// again. Decode to correct the display.
‘title’ => htmlspecialchars_decode($doc->label, ENT_QUOTES),
// These values are not required by the search module but are provided
// to give entity callbacks and themers more flexibility.
‘score’ => $doc->score,
‘snippets’ => $snippets,
‘snippet’ => $snippet,
‘fields’ => $fields,
‘entity_type’ => $doc->entity_type,
‘bundle’ => $bundle,
);

You can see here that this code is treating the results as objects.

‘title’ => htmlspecialchars_decode($doc->label, ENT_QUOTES),

However, if you look at the JSON response of Apache Solr 6.x you will see the results are mostly array elements not objects. Note the brackets.

 


"response":{"numFound":11871,"start":0,"docs":[
{
"id":"noim8l/node/3050",
"site":["http://yeahchemistry.com/"],
"hash":["noim8l"],
"entity_id":[3050],
"entity_type":["node"],
"bundle":["forum"],
"bundle_name":["Questions"],
"ss_language":["und"],
"path":["node/3050"],
"url":["http://yeahchemistry.com/questions/electron-orbitalsperiodic-trends-there-book-covers"],
"path_alias":["questions/electron-orbitalsperiodic-trends-there-book-covers"],
"label":["Electron Orbitals/Periodic Trends- is there a book that covers this"],
"content":[" We've started our Atomic Theory unit and have just completed electron configuration/orbital properties/shapes and have moved on to periodic trends (shielding, Zeff, etc.). Needless to say, I am thoroughly confused. Is there any website or book (besides my chem book--Zumdahl 5th Edition) that I can look to for help? I think that chemistry is really interesting, I am just struggling a lot. It's discouraging, but I'm not giving up! "],
"teaser":[" We've started our Atomic Theory unit and have just completed electron configuration/orbital properties/shapes and have moved on to periodic trends (shielding, Zeff, etc.). Needless to say, I am thoroughly confused. Is there any website or book (besides my chem book--Zumdahl 5th Edition) that I"],
"ss_name":["caileywailey"],
"tos_name":["caileywailey"],
"ss_name_formatted":["caileywailey"],
"tos_name_formatted":["caileywailey"],
"is_uid":[1784],
"bs_status":[1],

So it stands to reason that if the JSON element “label” is returning an array not an object then I should do the same in the Drupal module code.

$fields = (array) $doc;
// Define our url options. They depend on the document language.
$url_options = array(‘absolute’ => TRUE);
if (isset($doc->ss_language) && isset($languages[$doc->ss_language])) {
$url_options[‘language’] = $languages[$doc->ss_language];
}
$result = array(
// link is a required field, so handle it centrally.
‘link’ => url($doc->path[0], $url_options),
// template_preprocess_search_result() runs check_plain() on the title
// again. Decode to correct the display.
‘title’ => htmlspecialchars_decode($doc->label[0], ENT_QUOTES),
// These values are not required by the search module but are provided
// to give entity callbacks and themers more flexibility.
‘score’ => $doc->score,
‘snippets’ => $snippets,
‘snippet’ => $snippet,
‘fields’ => $fields,
‘entity_type’ => $doc->entity_type,
‘bundle’ => $bundle,
);

 

And now we are correctly populating the title variable and we get our search result title links.

 

To clean up all the errors you need to do this is many places. Let the errors lead you there, but you get the gist of it.

..