Friday, May 12, 2017

what is MQTT


MQTT on Juniper devices with Junos 16.1R1

For decades, syslog and snmp traps have been extensively used to receive event notifications from devices. It typically required an operator to configure the event category and severity level on the devices and where to send the events to.
Fast forward to today and thanks to a generic need for highly scalable and lightweight instant messaging needs, there is now MQTT (MQ Telemetry Transport), defined in an ISO standard (ISO/IEC PRF 20922): A client server publish/subscribe messaging transport protocol, run over TCP/IP.
Juniper introduced MQTT to Junos 16.1R1 as an integral part of the Juniper Extension Toolkit (JET). Chapter 2 in Juniper Extension Toolkit API Guide describes the notification API in detail, but where is the fun in reading when you can simply take a router for a spin!
First, make sure you run Junos 16.1R1 or higher, then enable MQTT with the following configuration:
$ ssh admin@vmx4
Last login: Sun Aug 28 09:21:41 2016 from 172.17.0.1
--- JUNOS 16.1R1.7 Kernel 64-bit  JNPR-10.1-20160624.329953_builder_stable_10
admin@vmx4> show configuration system services extension-service notification
port 1883;
allow-clients {
    address 0.0.0.0/0;
}
This pretty much opens up the router for worldwide attack. Access can be limited to IPv4/IPv6 networks/prefixes.
Second, install a suitable MQTT client on your host system. I opted here for one of the most basic, CLI based, clients there is on Linux: mosquitto-clients. Install it on Ubuntu with
sudo apt-get install mosquitto-clients
Third, launch mosquitto_sub against the router, either via IP address or hostname and wait for messages to trickle in. Instead of subscribing to specific event topics, we simply want everything (and in JSON format). This can be done using the wildcard ‘#’. See the man page for mqtt(7) for more details how to subscribe to specific events. Subscribing to everything via mosquitto_sub against a Junos device with notification extension-service turned on, will print all messages that it receives:
$ mosquitto_sub -h vmx4 -t \#
{
    "jet-event": {
        "event-id": "SYSTEM",
        "hostname": "vmx4",
        "time": "2016-08-28-09:21:41",
        "severity": "notice",
        "facility": "auth",
        "process-id": 80148,
        "process-name":"sshd",
        "message": "(pam_sm_acct_mgmt): DEBUG: PAM_USER: mwiget",
        "attributes": {
            "message": "(pam_sm_acct_mgmt): DEBUG: PAM_USER: mwiget"
        }
    }
}
{
    "jet-event": {
        "event-id": "SYSTEM",
        "hostname": "vmx4",
        "time": "2016-08-28-09:21:41",
        "severity": "info",
        "facility": "auth",
        "process-id": 80148,
        "process-name":"sshd",
        "message": "Accepted publickey for mwiget from 172.17.0.1 port 47968 ssh2: RSA 60:46:50:7b:d5:b7:6b:75:ba:a6:86:48:6d:44:63:e0",
        "attributes": {
            "message": "Accepted publickey for mwiget from 172.17.0.1 port 47968 ssh2: RSA 60:46:50:7b:d5:b7:6b:75:ba:a6:86:48:6d:44:63:e0"
        }
    }
}
{
    "jet-event": {
        "event-id": "UI_COMMIT_PROGRESS",
        "hostname": "vmx4",
        "time": "2016-08-28-09:30:51",
        "severity": "info",
        "facility": "interact",
        "process-id": 80151,
        "process-name":"mgd",
        "message": "UI_COMMIT_PROGRESS: Commit operation in progress: commit complete",
        "attributes": {
            "message": "commit complete"
        }
    }
}
{
    "jet-event": {
        "event-id": "UI_COMMIT_COMPLETED",
        "hostname": "vmx4",
        "time": "2016-08-28-09:30:51",
        "severity": "warn",
        "facility": "interact",
        "process-id": 80151,
        "process-name":"mgd",
        "message": "UI_COMMIT_COMPLETED: commit complete",
        "attributes": {

        }
    }
}
{
    "jet-event": {
        "event-id": "UI_COMMIT_PROGRESS",
        "hostname": "vmx4",
        "time": "2016-08-28-09:30:51",
        "severity": "info",
        "facility": "interact",
        "process-id": 80151,
        "process-name":"mgd",
        "message": "UI_COMMIT_PROGRESS: Commit operation in progress: signaling 'Alarm control process', pid 4180, signal 30, status 0 with notification errors enabled",
        "attributes": {
            "message": "signaling 'Alarm control process', pid 4180, signal 30, status 0 with notification errors enabled"
        }
    }
}
{
    "jet-event": {
        "event-id": "PIC",
        "hostname": "vmx4",
        "time": "2016-08-28-09:30:51",
        "severity": "debug",
        "facility": "pfe",
        "process-id": 0,
        "process-name":"UNKNOWN",
        "message": "fpc0   FPC slot: 0, DISABLE Delay: 5, Frequency: 60\n",
        "attributes": {
            "message": "fpc0   FPC slot: 0, DISABLE Delay: 5, Frequency: 60\n"
        }
    }
}
{
    "jet-event": {
        "event-id": "SNMP_TRAP_LINK_DOWN",
        "hostname": "vmx4",
        "time": "2016-08-28-09:30:51",
        "severity": "warn",
        "facility": "daemon",
        "process-id": 4086,
        "process-name":"mib2d",
        "message": "SNMP_TRAP_LINK_DOWN: ifIndex 529, ifAdminStatus down(2), ifOperStatus down(2), ifName ge-0\/0\/1.0",
        "attributes": {
            "snmp-interface-index": "529",
            "admin-status": "down(2)",
            "operational-status": "down(2)",
            "interface-name": "ge-0\/0\/1.0"
        }
    }
}
Physical Interface (IFD), Logical Interface (IFL), Family (IFF), Address, Firwall, Route, Route-table and Syslog are Event Topics one can subscribe to. See JET Notification API Overview from http://www.juniper.net/techpubs/ for details about these Junos Event Topics.




https://marcelwiget.wordpress.com/2016/08/28/mqtt-on-juniper-devices-with-junos-16-1r1/#more-61
https://www.youtube.com/watch?v=EIxdz-2rhLs

JET API from juniper

JET API of Juniper router MX:
For decades, syslog and snmp traps have been extensively used to receive event notifications from devices. It typically required an operator to configure the event category and severity level on the devices and where to send the events to.
Fast forward to today and thanks to a generic need for highly scalable and lightweight instant messaging needs, there is now MQTT (MQ Telemetry Transport), defined in an ISO standard (ISO/IEC PRF 20922): A client server publish/subscribe messaging transport protocol, run over TCP/IP.
Juniper introduced MQTT to Junos 16.1R1 as an integral part of the Juniper Extension Toolkit (JET). Chapter 2 in Juniper Extension Toolkit API Guide describes the notification API in detail, but where is the fun in reading when you can simply take a router for a spin!

Reference:

What is Thrift

Thrift:
Point 1:

An RPC framework in general is a set of tools that enable the programmer to call a piece of code in a remote process, be it on a different machine or just another process on the same machine.

In the particular case of Apache Thrift, we talk about a framework designed to be efficient, and available across both OS platforms and programming languages. Additionally, you have some flexibility regarding transports (such as sockets, pipes, etc) and protocols (binary, JSON, even compressed), plus some more options like SSL or SASL support.

For example, you may set up a server on a Linux machine, written in C++ which offers some service to the world through a JSON-based protocol over HTTP. This service may be called by a client program written in Python, running on a Windows machine. The code for both server and client is generated from a Thrift IDL file. To get it running, you basically have to add only the intended program logic and put all the pieces together.

The single best reference for Apache Thrift is still the Apache Thrift Whitepaper. Although slightly outdated in some of the details, the underling concepts are still valid. Another good read is Diwaker Gupta's "Missing Guide", and last not least the forthcoming book from Randy Abernethy.

For beginners, I would recommend to start with the Apache Thrift tutorial suite, these examples show a lot of the core features. If you run into questions, you are welcome to ask here on SO, or on the Thrift mailing lists.

Point 2:
RPC (Remote Procedure Call) is like calling a function, only that it is present remotely on a different server as a service. A service exposes many such functions/procedure to its client. And client requires some way to know what are the functions/procedures exposed by this service and what are their parameters.

This is where Apache Thrift comes in. It has its own "Interface Definition Language" (IDL). In this language you define what are the functions and what are their parameters. And then use Thrift compiler to generate corresponding code for any language of your choice. What this means, is that you can implement a function in java, host it on a server and then remotely call it from python.

Important work a framework like Thrift does is this -
  1. Provide a language agnostic Interface Definition Language
  2. A Compiler to compile this IDL to produce client and server code (in same or separate language as required)
  3. Compiler generated client code exposes a stub interfaces for these functions. The stub code converts the parameters passed to the function into a binary (serialized) format that can be transported on wire over network. This process is called marshaling. The generated client code never has the actual implementation of the function, hence its called a stub.
  4. At server, the developer use the Compiler generated server code, to actually implement these functions (i.e. write the actual functionality of the function). Generated server side code receives the binary encoded message from client, converts them back to the corresponding language objects and passes it to the developer implemented function. This is called as unmarshaling. In java for example the Compiler generated server code would be Interface that the developer will implement and also various other classes.
  5. Similarly the result of a function is converted to binary and send to client.

For parameters to the function, IDL defines its own set of data structure types like List, Map, Struct or Classes apart from native types like Int, String, Boolean, etc. These are then mapped to corresponding language implementations.

Thrift is similar to SOAP and CORBA. Since they both are used for RPC and provide their own IDL. CORBA and SOAP generally also has a service discovery broker as a middleware for exposing functions/methods to client. For thrift, we normally use Zookeeper for service discovery.

REST is different, because it does not have IDL, And uses HTTP Methods like GET, PUT and url patterns to call a remote function and pass parameters. Using HTTP methods and url semantics makes it also language agnostic.

Messaging queue is entirely different. Because it is mostly used in Publish/Subscribe model. Whereas RPC is Client/Server model.

In Publish/Subscribe, multiple publishers sends/add a serialized message on a queue. The message format is defined by the publisher and has complete control of it. Their definition is semantically associated to the queue on which they are published, but there is no strict checking for their structure. Subscriber, then knowing the kind of the message a queue will have, subscribes for those messages. Publishers dont know who are the client, and Subscribers dont know who are the producers of the message. They only know what kind of message to publish or consume respectively from a queue. The Publisher and Subscriber is responsible for knowing the right serializer and deserializer.
This is different in Client/Server RPC, since Client knows (in strict sense) what to pass and Server defines it. And also whom to pass.

Other library similar to Thrift is Protobuf and Avro.
Finagle by Twitter is one way to create Thrift based service, it also have support for Protobuf.

point 3:
There's a lot of repeated work you have to do when you're writing a server - primarily designing a protocol and writing code to serialize and deserialize messages on the protocol, but also dealing with sockets and managing concurrency, and writing clients in many languages. Thrift automatically does all of this, given a description of the functions you want to expose from your server to clients. It's also useful for serializing data on disk or into shared memory (where many of the same problems come up).

Application :
Juniper JET is using thrift
https://www.juniper.net/documentation/en_US/jet1.0/topics/concept/jet-overview.html


Reference:
http://stackoverflow.com/questions/20653240/what-is-rpc-framework-and-apache-thrift
http://thrift.apache.org/static/files/thrift-20070401.pdf
http://diwakergupta.github.io/thrift-missing-guide/thrift.pdf
https://www.quora.com/In-simple-terms-what-is-Thrift-software-framework-and-what-does-it-do

Demo:
https://www.youtube.com/watch?v=NK6hz2JM89w

Wednesday, May 10, 2017

Juniper Line Card types

I'm going to build on this post as I come across each of the different types but it can get very confusing talking about the different types of Juniper Line Cards, so here is a reference:

DPC - Dense Port Concentrator
ichip based cards which are available for the MX series routers
They come in 3 variations:
DPCE-R - Routing and Switching, operate as complete L3 router or Full L2 switch
DPCE-X - Limited Scale L3. Cost optimized line case
DPCE-Q - Enhanced Queueing, up to 64,000 queues per card. Per VLAN queueing.

MPC - Modular Port Concentrator
trio based cards for the MX series routers. They support full LS, L2 and application services. MICs are used inside MPCs to provide interfaces.
MPC1 - 32k IFL, port queues, 30Gbps
MPC2 - 64k IFL, port queues, 60Gbps
MPC1-Q - 32k IFL, VLAN queues, 128k I/E queues, 30Gbps
MPC2-Q - 64k IFL, VLAN queues, 256k I/E queues, 60 Gbps
MPC2-E-Q - 64k IFL, VLAN queues, 512k queues, 60Gbps

MX-FPC - MX Flexible Port Concentrator
These are used to add non-Ethernet interfaces to an MX series chassis. They take up 2 slots and have reduced performance (2.5 Gbps Type 2 or 10Gbps Type 3)

More to come as I come across them...

Reference: http://twhittle1.blogspot.in/2012/08/juniper-line-card-types-dpc-mpc-etc.html

Thursday, May 4, 2017

What is DNS server and how its works



DNS server and its look up:

Domain Name Servers (DNS) are the Internet's equivalent of a phone book. They maintain a directory of domain names and translate them to Internet Protocol (IP) addresses.

This is necessary because, although domain names are easy for people to remember, computers or machines, access websites based on IP addresses. 

Information from all the domain name servers across the Internet are gathered together and housed at the Central Registry. Host companies and Internet Service Providers interact with the Central Registry on a regular schedule to get updated DNS information. 

When you type in a web address, e.g., www.jimsbikes.com, your Internet Service Provider views the DNS associated with the domain name, translates it into a machine friendly IP address (for example 216.168.224.70 is the IP for jimsbikes.com) and directs your Internet connection to the correct website. 

After you register a new domain name or when you update the DNS servers on your domain name, it usually takes about 12-36 hours for the domain name servers world-wide to be updated and able to access the information. This 36-hour period is referred to as propagation.



The Domain Name System (aka DNS) is used to resolve human-readable hostnames like www.Dyn.com into machine-readable IP addresses like 204.13.248.115. DNS also provides other information about domain names, such as mail services.
But why is DNS important? How does it work? What else should you know?

Why is DNS important?

DNS is like a phone book for the Internet. If you know a person’s name but don’t know their telephone number, you can simply look it up in a phone book. DNS provides this same service to the Internet.
When you visit http://dyn.com in a browser, your computer uses DNS to retrieve the website’s IP address of 204.13.248.115. Without DNS, you would only be able to visit our website (or any website) by visiting its IP address directly, such as http://204.13.248.115.

How does DNS work?

What is DNS? - DNS Flow Chart
When you visit a domain such as dyn.com, your computer follows a series of steps to turn the human-readable web address into a machine-readable IP address. This happens every time you use a domain name, whether you are viewing websites, sending email or listening to Internet radio stations like Pandora.
Step 1: Request information
The process begins when you ask your computer to resolve a hostname, such as visiting http://dyn.com. The first place your computer looks is its local DNS cache, which stores information that your computer has recently retrieved.
If your computer doesn’t already know the answer, it needs to perform a DNS query to find out.
Step 2: Ask the recursive DNS servers
If the information is not stored locally, your computer queries (contacts) your ISP’s recursive DNS servers. These specialized computers perform the legwork of a DNS query on your behalf. Recursive servers have their own caches, so the process usually ends here and the information is returned to the user.
Step 3: Ask the root nameservers
If the recursive servers don’t have the answer, they query the root nameservers. A nameserveris a computer that answers questions about domain names, such as IP addresses. The thirteen root nameservers act as a kind of telephone switchboard for DNS. They don’t know the answer, but they can direct our query to someone that knows where to find it.
Step 4: Ask the TLD nameservers
The root nameservers will look at the first part of our request, reading from right to left — www.dyn.com — and direct our query to the Top-Level Domain (TLD) nameservers for .com. Each TLD, such as .com.org, and .us, have their own set of nameservers, which act like a receptionist for each TLD. These servers don’t have the information we need, but they can refer us directly to the servers that do have the information.
Step 5: Ask the authoritative DNS servers
The TLD nameservers review the next part of our request — www.dyn.com — and direct our query to the nameservers responsible for this specific domain. These authoritative nameservers are responsible for knowing all the information about a specific domain, which are stored in DNS records. There are many types of records, which each contain a different kind of information. In this example, we want to know the IP address for www.dyndns.com, so we ask the authoritative nameserver for the Address Record (A).
Step 6: Retrieve the record
The recursive server retrieves the A record for dyn.com from the authoritative nameservers and stores the record in its local cache. If anyone else requests the host record for dyn.com, the recursive servers will already have the answer and will not need to go through the lookup process again. All records have a time-to-live value, which is like an expiration date. After a while, the recursive server will need to ask for a new copy of the record to make sure the information doesn’t become out-of-date.
Step 7: Receive the answer
Armed with the answer, recursive server returns the A record back to your computer. Your computer stores the record in its cache, reads the IP address from the record, then passes this information to your browser. The browser then opens a connection to the webserver and receives the website.
This entire process, from start to finish, takes only milliseconds to complete.


Reference:
http://dyn.com/blog/dns-why-its-important-how-it-works/
http://www.networksolutions.com/support/what-is-a-domain-name-server-dns-and-how-does-it-work/