The first thing I did- and this is very important- was to look at a lot of Myst art. Screenshots from the game, graphics from Myst-related websites, concept art, game guides, fan art... not only is it good research, allowing you to discover different shapes, colors, and effects that are common to the things you're trying to emulate, it also helps get you in the mood and mindset of the people whose style you're trying to mimic.
The first step in the actual design process was to set up my graphics. I created the images I wanted in Illustrator- at this point I did not worry about the colors or the jagged edges- I simply created the basic shapes. I made heavy use of the pathfinder tool- if you're not familiar with this you'll want to mess around with it a lot as it is incredibly useful.
I created the border image first, deciding to worry about the illustrations for the card last. This was just a matter of personal preference, it was what I felt like doing at the time. Always listen to yourself- never force yourself to do something you don't want to on a personal project, or it will look forced. At this point, the word "Ahyoheek" in D'ni was created using the D'ni font from the Guild of Linguists. Check it out on Google.
Once I was done with my images, I copied the shapes from Illustrator and pasted them into Photoshop. My Photoshop file was created from a template offered by The Game Crafter. When Photoshop asked how I wanted to paste the content, I chose the button for "Path." When the path is first imported, you can hit command+T (or Edit>Free Transform) to resize the path appropriately. This is important when using The Game Crafter, as all of your artwork must fit within certain boundaries in order to be properly printed.
I then created a custom Photoshop brush. This is leaving out a lot of important points, but as for a crash course: Create a Photoshop file that is 200x200 pixels. Fill with white. Use a standard brush tool to draw an image in black on the white background. Select all, then choose Edit>Define Brush Preset. Once this preset is defined you can use it in other files, and the "Brush" panel gives you more options you can adjust, such as random jittering of the size of the brushstroke. Don't ignore this part, it is very important to get a hand-drawn look!
After creating my brush I used it to stroke the path. Select the brush tool, in the "Paths" panel select your path, and then select the small, hollow circle button from the bottom of the panel. You'll want to make sure that you are set to a blank layer at this point, this will make sure that the linework becomes its own layer. At this point I added some manual detail to the linework using the brush tool. I filled in parts of the line that didn't turn out as I wanted them, and also added the flourishes and uneven strokes to the D'ni font.
You may want to copy different parts of your design into different paths, then stroke them with different thicknesses of brush. This creates more interest in your design, and appears as though the artist used varying pressure on the pen to represent different levels of detail. Compare the lines around the border knot design to the lines enclosing said design to see this technique in action.
To create the paper texture, I filled my background layer with a yellow color, then used the same yellow color in a brush. I selected the DP Crackle brush from DP Brushes, then used shape dynamics from the brush panel to jitter the size and rotation. I didn't want to page to look crackled, but this is a good brush for creating filaments. I set the brush to about 5% Multiply, then scribbled all over the card. I changed the brush size slightly, scribbled again, changed it to 10% Screen, scribbled again... you get the idea. You'll want to create layer after layer of filaments all over the file. You might also want to mess around with other brushes to add dust flecks, mold spots, wrinkles, tears... everything you can think of. Create a story for your piece of paper- what was it made from? Who made it? How did it get abused through its life? Was it exposed to salt water? Left to rot on a cliffside? Exposed to radioactive waste?
The color was added by duplicating the linework layer, setting it to multiply, and filling the shapes like a coloring book. I made change after change to this to get just the right colors- I wanted them to be bright and colorful- yet dull enough to appear hand-crafted and aged.
I used the "Animation" panel to create each individual card as an animated frame. While I did not intend to output it as an animation, this made it easy to export them all at the same time. Once I was finished, I selected File>Save for Web & Devices. I selected to output it as a Png-24, and clicked save. Since Photoshop does not support animated Png files (and most web browsers don't either, for that matter,) they were automatically saved as a number of separate files. This may not have been as important for this 4-card project, but it really comes in handy if you're making, say, a custom 52-card standard deck. Which I have. Before realizing I could do this. It was an unpleasant experience.
I got my art-student girlfriend Chelsea to give me feedback for the colors throughout the entire process- I am dichromatic (see everything in shades of red and blue,) so I'm at a disadvantage for that. Never be afraid to recognize your own flaws and account for them. Being a good artist or designer is not about being perfect- it's about knowing how to work with, around, and through your imperfections.