Encoder is a digital circuit that converts an active signal into a coded output signal.
General idea of encoder is illustrated in following block diagram
There are n input lines, only one of which is active. Encoder converts this active input to the binary coded output with m bits.
When button 3 is pressed, the C and D OR gates have high inputs, therefore output is ABCD=0011 i.e. decimal 3 is converted into its equivalent binary number 0011.
(Verify this conversion using online calculator by clicking on http://www.easycalculation.com/binary-converter.php)
Similarly, if you press button 5, the output will be ABCD=0101
and for switch 9, we get ABCD=1001
The IC-74147 of Encoder:
The above figure shows the pinout diagram for IC-74147, a decimal-to-BCD encoder.Pins 1 to 5 and 10 to 13 are for X1 to X9 decimal inputs and pins 14,6,7 and 9 are for BCD output. Pin 16 is for supply voltage and pin 8 is grounded. Notice bubble at input and output, which indicated active-low signal.
When all X inputs are high, all outputs are high. When X9 is low, the ABCD output is LHHL or 0110 which is binary equivalent of 9. When X8 is only low input, ABCD is LHHH or 0111 which is binary equivalent of 8.
IC-74147 is called priority encoder because it gives priority to highest order input. That means if all inputs from X1 to X9 are low, it gives priority to highest order 9 which is encoded into out as LHHL. i.e. X9 has priority over all others. When X9 is high it gives priority to X8 and gets encoded if it is low.